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EXECUTIVE  SUMMARY 


This  report  documents  the  results  of  a  series  of  studies  of  the  intact  of  Acquisition  Reform  on 
the  process  of  engineering  systems  used  aboard  ships  during  operational  deployments.  The 
initial  study  focused  on  the  question:  What  role  should  the  in-house  naval  engineering 
community  play  in  acquiring  and  supporting  future  systems,  and  specifically,  what  is  the  mpact 
on  the  engineering  process  if  commercial  products  are  used  versus  the  traditional  Navy  products? 
Subsequence  studies  investigated  various  aspects  of  this  compound  question. 

A  fiindamftntfll  feature  of  the  traditional  acquisition  process  was  product  certijication;  that  is,  the 
risk  to  the  user  was  known,  understood,  and  minimized.  This  concept  of  certification  carries 
with  it  both  legislative  and  engineering  responsibility.  Certifying  a  system  means  qualifying  the 
risk  in  using  it  by: 

•  Defining  risk  criteria. 

•  Verifying  the  system  meets  the  definition. 

•  Testifying  to  the  fidelity  of  the  Verification- 

Certifying  shipboard  systems  is  an  integral  part  of  acquisition.  The  ashore  engineering  and 
management  infrastructure  is  accountable  fi>r  ensuring  that  the  material  elements  of  a  deployed 
command  are  mission  reafy.  To  be  mission  ready,  shipboard  i^stems  must  be  available, 
reliable,  and  maintainable  over  their  life  cycle.  Certification  reflects  knowledge  of  the  system 
design  and  how  it  is  maintained.  The  act  of  certification  inq>licitfy  states  that: 

•  The  intended  use  of  the  system  is  understood. 

•  The  environment  the  system  will  operate  in  is  known. 

•  The  system  design  is  consistent  with  the  operating  environment  and  intended  use. 

•  The  system  was  implemented  consistent  with  its  design. 

•  The  capabilities  and  limitations  of  the  system  implementation  are  known. 

•  The  maintenance  process  will  preserve  the  design  inqjlementation. 

•  Design  changes  (even  the  most  minor  ones)  will  require  recertification. 

•  The  i^stem  was  properly  installed  aboard  ship. 

•  The  ship’s  crew  has  correct  and  sufScient  information  to  operate  and  maintain  the 
system  at  sea. 

The  fundamental  concept  of  developing,  producing,  and  supporting  shipboard  systems  is 
represented  by  an  input-output  model  of  the  core  technical  process  for  engineering  and 
management  of  military  systems.  Inputs  to  the  process  are  derived  from  policy,  mission  needs, 
and  resources  and  the  outputs  of  the  process  are  products  to  be  used  by  flie  operational  maritime 
force.  Acquisition  Reform  has  introduced  inportant  changes  in  the  input  to  the  core  technical 
process  for  shipboard  systems,  particularly  for  weapons  that  contain  conputer  controlled  fire 
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control  loops.  These  changes  have  destabilized  the  process  for  some  shipboard  ^sterns  in  terms 
of  qualifying  their  risk  to  the  user  and  certifying  them  as  mission  ready.  To  reconstitute  a  stable, 
cost  effective  process,  it  is  necessary  to  understand  system  certification  in  terms  of  the  core 
technical  process  and  the  changes  in  input  introduced  by  Acquisition  Reform.  There  are 
organizational  and  acquisition  roadblocks  to  a  smooth  transition  to  a  new  core  technical  process. 

Overcoming  organizational  roadblocks  requires  dealing  aggressively  with  commercial  products 
that  have  short  lifetimes  by: 

•  Integrating  the  roles  of  development,  production,  and  support. 

•  Creating  buffers  to  rapid  changes  and  the  obsolescence  of  commercial  items. 

In  addition,  nondisclosure  of  information  about  proprietary  commercial  products  is  a  special 
problem  in  establishing  a  new  core  technical  process.  Nondisclosure  and  short  lifetime  issues 
form  interrelated  roadblocks  to  acquisition,  and  overcoming  one  may  help  overcome  the  other. 
And,  both  are  roadblocks  to  qualifying  risk  and  ultimately  certifying  systems  as  mission  ready. 
Alternative  methods  for  government  use  and  protection  of  proprietary  products  are  needed,  and 
'One  approach  could  involve  establishing  a  second  source  for  selected  commercial  products. 
Institutionalizing  a  new  core  technical  process  involves  management,  engineering,  and 
legislative  issues.  Consideration  should  be  given  to  review  and  possible  modification  of  the 
United  States  Code  and  the  DoD  acquisition  instructions  based  upon  it. 

The  study  reported  here  is  not  all  enconqjassing.  Not  addressed  are  systems  used  ashore  for 
such  purposes  as  simulation,  anafysis,  or  training.  Nor  is  the  use  of  commercial  products  in  hull 
and  machinery  systems  considered.  The  issue  of  electromagnetic  interference  is  not  included 
although  it  is  a  growing  problem  due  to  the  increased  use  of  communication  equ^ment  aboard 
ship.  Although  standards  are  necessary  for  certification,  this  study  did  not  review  the  residual 
Military  Standards  and  available  conunercial  standards.  Work  following  this  study  should 
address  certification  at  the  total  ship  level  and  enconqrass  hull,  machinery,  weapons,  and 
communications. 
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1.0  INTRODUCTION 


This  report  docximents  a  series  of  studies  that  were  done  to  assess  the  impact  of  Acquisition 
Reform  on  the  process  of  engineering  systems  used  aboard  ships  during  operational 
deployments.  Acquisition  Reform  advocates  using  commercial  products  and  “managing  the 
suppliers  not  the  supplies.”  This  new  approach  raises  a  conqjlicated  question:  What  shoidd  be 
the  role  of  the  in-house  naval  engineering  community  in  acquiring  and  supporting  ftoe 
systems,  and  specifically,  what  is  the  impact  on  the  engineering  process  if  commercial  products 
are  used  in  place  of  tra^tional  Navy  products? 

The  challenge  in  answering  this  con^licated  question  was  to  unravel  the  interrelationship  of 
organization,  process,  and  product  to  identify  those  characteristics  that  must  be  preserved  in 
going  fi-om  the  old  traditional  approach  to  the  new  Acquisition  Reform  approacL  piere  were 
many  differences  that  arose  in  comparing  the  old  with  the  new,  and  each  of  these  differences  was 
perceived  at  one  time  or  another  to  be  the  issue  in  using  commercial  products.  Differences  in 
approach  involved  almost  everything  that  characterized  process  and  product;  for  example, 
configuration  management,  hardness,  maintenance,  interfeces,  security,  performance,  testii^, 
requirements,  and  documentation.  Identifying  differences  as  issues  only  succeeded  in  causing 
the  proponents  of  Acquisition  Reform  to  view  the  results  as  conq)laints  about  change  and  a 
desire  to  continue  the  status  quo.  This  criticism  led  to  focusing  on  why,  in  the  traditional 
approach,  things  had  been  done  the  way  they  had.  Was  there  an  underlying  motive,  or  was  it 
simply  a  way  to  create  jobs  and  bureaucracy?  It  was  concluded  that,  although  there  was  ample 
bureaucracy  surroimding  it,  there  was  an  underlying  motive  for  the  traditional  process.  Md  that 
motive  was  to  ensure  products  would  work  as  eiqpected  when  used  under  military  conditions. 

A  singular  exanqile  of  the  desire  that  products  work  as  ejqiected  under  mihtary  conditions 
occurred  during  the  Normandy  invasion  June  6, 1944.  Receiving  news  that  the  invasion  was  on, 
several  American  fectories  momentarily  shut  down  assembly  lines  so  the  workers  could  pray  that 
the  equipment  they  had  supplied  would  not  feil  the  Allied  troops  who  were  perceived  at  that 
moment  to  be  engaged  in  combat.  Since  1944  some  nulitaiy  products  have  grown  into  complex 
computer  controlled  systems.  However,  the  desire  that  military  systems  not  feil  those  vAio 
depend  on  them  has  not  changed,  nor  can  it  change.  Satisfying  this  desire  was  the  underlying 
motive  and  the  legacy  of  the  traifeional  acquisition  process.  The  results  of  the  series  of  studies 
reported  here  have  identified  this  fundamental  feature  of  the  traditional  acquisition  process  as 
certification'^  that  is,  the  risk  to  the  user  is  known,  understood,  and  minimized.  This  concept  of 
certification  carries  with  it  both  legislative  and  engineering  responsibility.  The  government  is 
ultimately  responsible  to  its  people  for  the  actions  of  mihtary  commanders  and  for  the  sysftems 
they  use.  Certifying  shipboard  systems  is  an  integral  part  of  acquiring  them,  and  the 
management  and  engineering  infrastructure  is  accountable  for  ensuring  that  the  material 
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elements  of  an  afloat  command  are  mission  ret^ — ^available,  reliable,  and  maintainable. 
Although  factory  workers  in  private  industry  may  never  again  need  to  pray  that  their  products 
not  M  the  troops,  it  will  always  remain  the  responsibility  of  the  government  to  establish  and 
maintain  an  infrastructure  to  ensure  that  the  risks  of  such  inures  are  minimized. 

The  studies  reported  here  considered  the  intact  of  Acquisition  Reform  on  engineering  Navy 
systems  intended  for  deployment  at  sea  Systems  used  ashore  for  other  purpose  such  as 
simulation,  analysis,  or  training  were  not  specifically  considered.  Other  studies  have  been 
conducted  on  the  qualification  of  shore  based  acquisition  support  systems  [1].  And  the  Joint 
Chiefr  of  Staff  have  assumed  the  responsibility  for  interoperability  requirements  certification  [2]. 

This  report  describes  two  approaches  to  engineering  complex,  automated  weapon  systems: 

•  The  traditional  approach  using  Navy  qualified  components,  and 

•  The  Acquisition  Reform  approach  using  components  commercially  available  off-the- 
shelf 

-  Both  these  approaches  may  be  regarded  as  extreme.  The  traditional  approach  adapted  to  an 
environment  in  which  the  applicable  technologies  were  developed  and  controlled  by  the 
Department  of  Defense.  When  this  control  was  made  inqwssible  by  the  rapid  growth  of  digital 
technology  in  the  commercial  market,  a  change  in  process  was  needed.  The  traditional  approach 
had  been  successfully  used  for  decades,  but  its  layered  bureaucracy  could  not  easily  purge 
unnecessary  and  outdated  elements  of  the  process,  and  it  evolved  at  its  own  rate,  not  in  response 
to  the  conqjetitive  commercial  market.  In  contrast,  the  reform  approach,  in  use  for  less  than  a 
decade  and  untested  for  large  complex  systems,  avoids  the  ponderous  military  acquisition 
process  and  depends  on  products  that  are  developed  rapidly  in  response  to  the  competitive 
commercial  market.  The  reform  approach  assumes  commercial  products  can  be  readily  used  as 
unmodified  parts  of  military  systems. 

These  two  approaches — one  slow,  plodding,  and  methodical;  the  other  fast,  adaptable,  and 
entrepreneurial — ^have  a  common  purpose;  to  produce  conq)lex  shipboard  systems  that  work  as 
e3q)ected  and  can  be  supported  at  sea.  The  remainder  of  this  report  provides  results  of  studies 
conducted  off-and-on  from  1995  through  2000  to  investigate  the  basic  process  of  providing  and 
supporting  shipboard  systems. 
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2.0  THE  CORE  TECHNICAL  PROCESS 


The  fiinHamfttitfll  concept  for  developing,  producing,  and  supporting  shipboard  systems  is  the 
input-output  model  that  represents  the  core  technical  process  for  engineering  mihtary  systems  as 
shown  in  Figure  1.  Inputs  to  the  process  are  derived  from  policy,  mission  needs,  and  resources, 
and  the  ouQ)uts  of  the  process  are  products  to  be  used  by  the  operational  maritime  force.  Where: 

•  Policy  is  the  result  of  legislation  that  is  expressed  in  the  United  States  Code  and  also 
interpreted  by  instructions  issued  by  the  Office  of  Secretary  of  Defense,  the  Office  of  the 
Secretary  of  the  Navy,  and  the  Chief  ofNaval  Operations.  Policy  can  include  priority 
and  cost. 

•  Mission  needs  are  derived  from  national  and  international  policy  and  from  operational 
experiences  of  deployed  forces  and  can  include  operational  performance  and  schedule. 

•  Resources  include  frinding,  manpower  levels,  and  regulatory  power. 

•  Products  are  certified  shipboard  systems  and  all  that  goes  with  them,  such  as 
documentation,  e5q)endables,  and  spare  parts. 


Crae  Tedmical  Process 


Pdicy, 

Mssioo  Needs, 
&Resources 
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Input 


Contrd 


^stem  DevelqHuent, 
Production,  feSigycrt 
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FIGURE  1.  INPUT-OUTPUT  MODEL 

The  core  technical  process  is  not  self  supporting,  it  is  sustained  by  irq)ut  resources.  The  primary 
purpose  of  the  process  is  to  provide  the  desired  output,  and  ideally,  this  should  be  done  with  the 
least  amoimt  of  resource  input.  The  cost  of  operating  the  process  is  determined  by  the  process 
itself^  not  by  limiting  input  resources. 

In  Figure  1,  input  is  converted  to  output  by  the  core  technical  process.  Ideally  the  process  is 
linear  and  stable;  that  is,  the  output  is  proportional  to  the  input  for  a  wide  range  of  inputs.  The 
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process  will  have  an  input-to-output  response  time  that  determines  the  interval  between  the 
desire  for  a  product  and  delivery  of  the  product.  Response  time  will  depend  on  inputs  and  on 
the  process  itself.  Ideally  the  process  is  designed  so  that  the  time  required  to  fill  a  need  is 
minimized.  Response  time  is  an  inherent  characteristic  of  the  process  and  it  cannot  be  changed 
simply  by  changing  the  rate  of  change  of  the  input. 

The  actual  process  represented  in  Figure  1  is  a  conqjlex  of  interrelated  government  and  industry 
organizations  and  fecilities,  and  there  are  multiple  inputs  and  outputs  at  dijfferent  points  within 
the  conqjlex.  Arbitrary  changes  can  occur  in  the  inputs  inducing  variance  that  can  cause  the 
process  response  to  become  nonlinear  or  unstable.  Instability  is  reflected  by  products  that  are 
inadequate,  too  expensive,  or  appear  too  slowly.  When  this  occurs,  the  core  technical  process 
will  adapt,  if  possible,  and  a  motMed  process  emerge  that  is  responsive,  linear,  stable,  and  cost 
effective.  Adaptation  typically  involves  a  learned  tinkering  with  the  controls  to  chaise  the 
process  to  the  desired  behavior.  Acquisition  Reform  has  introduced  important  changes  in  the 
input  to  the  core  technical  process  for  shipboard  systems,  particularly  for  weapons  that  contain 
'  conqjuter  controlled  fire  control  loops.  These  changes  have  destabilized  the  process  for  some 
shipboard  ^sterns  in  terms  of  qualifying  their  risk  to  the  user  and  certifying  them  as  mission 
ready.  To  reconstitute  a  stable,  cost  effective  process,  it  is  necessary  to  xmderstand  system 
certification  in  terms  of  the  core  technical  process  and  changes  in  input  introduced  by 
Acquisition  Reform. 


2.1  ACQUISITION  REFORM  AND  SYSTEM  CERTIFICATION 

By  the  early  1990s  the  basic  motive  of  the  tradition  acquisition  process — quality  control — had 
become  buried  imder  years  of  acquisition  process  evolution  and,  to  proponents  of  reform,  the 
onfy  visible  characteristics  of  engineering  military  systems  were  expense,  inefficiency,  and 
bureaucracy.*  This  perception  was  intensified  by  revolutionary  changes  in  office  automation 
and  personal  communications  brought  about  by  the  widespread  use  of  digital  conputers.  Digital 
technology  developed  by  the  Department  of  Defense  in  earlier  decades  had  been  slowly 
transferred  to  private  industry  and,  by  the  1980s,  inexpensive  conmercial  conqruters  were 
readily  available.  The  Navy  standard  corr5)uter,  the  AN/UYK-43,  that  was  deployed  on  newly 
constructed  ships  starting  in  1983  was,  in  terms  of  processor  speed  and  memory,  less  capable 
than  commercial  corr^uters  that  were  selling  daily  for  a  few  hundred  dollars.  To  the  advocates 
of  reform,  the  traditional  process  was  a  roadblock  to  acquiring  the  latest  technology 
inexpensively  fi'om  commercial  vendors  [3].  Indeed,  a  commercial  computer  with  more  than  ten 
times  the  speed  and  memory  could  be  bought  over-the-counter  for  less  than  one  tenth  the  cost  of 
the  AN/UYK-43.  Acquisition  comparisons  were  usually  limited  to  speed,  memory,  and  cost, 
and  the  concepts  of  quality  assurance  and  certification  were  not  visible.  The  end  of  the  Cold 
War  with  the  Soviet  Union  offered  an  opportunity  for  reform.  With  the  threat  of  war  ended, 
reform  could  be  experimented  with  in  a  peacetime  environment  where  a  new  core  technical 
process  based  on  commercial  products  could  be  evolved  in  relative  military  security. 

*  The  General  Accounting  Office  has  declared  that  the  quality  assurance  goal  is  “. .  .to  ensure 
products  perform  the  way  they  are  supposed  to...”  and  perceived  the  DoD’s  quality  assurance 
practices  as  adding  unnecessary  overhead  (letter  report,  08/26/96,  GAO/NSIAD-96-162). 
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Examples  used  to  support  die  Acquisition  Reform  movement  are  typically  based  on  the 
procurement  of  non  system  products  such  as  ant  bait  or  chairs,  or,  for  exanple,  conparisons  of 
personal  con^uter  speed  and  memory  with  military  system  conqxinents  such  as  the  ANAJYK- 
43.  In  these  con^arisons,  the  traditional  military  approach  is  shown  to  be  fer  more  costly  than 
the  commercial  or  private  approach.  These  comparisons  are  true,  the  costs  are  radically 
different,  and  the  traditional  acquisition  process  has  likely  been  inefficient  in  acquiring  products. 
The  Department  of  Defense  process  for  acquiring  ant  bait,  chairs,  and  other  eiqpendable  and  non 
expendable  commercially  available  products  may  indeed  be  streamlined  and  a  real  cost  savings 
realized.  Unfortunatefy,  comparisons  are  limited  to  such  sinqile  products  and  the  process  for 
acquiring  conplex  military  systems  such  as  a  surfece  combatant  has  no  equiv^nt  counterpart  in 
the  private  sector.  “Systems”  are  typically  two  or  more  physically  and  functionally  related  parts 
that  are  interconnected  to  form  a  “machine”  capable  of  converting  ii^ut  to  output.  Con^iuter 
controlled  sj^ems  have  the  added  conplexity  of  a  concputer  program  that  executes  in  real  time 
to  automatically  operate  the  machine. 

Conq)arison  of  military  weapon  systems  with  commercial  systems  can  be  made  to  a  degree.  The 
typical  exanq)les  given  are  banking  and  automated  chemical  plants. 

•  Banking  and  weapons  are  argued  to  be  similar  in  that  Mure  of  either  while  in  use  will 
result  in  financial  loss.  Banking  systems  are  designed  for  the  accurate  transfer  and  accounting  of 
money.  Weapon  systems  are  designed  to  destroy  property  and  take  human  life.  The  destruction 
of  property  and  life  in  combat  does  not  conpare  one-for-one  with  money  lost  in  a  Mlty  financial 
transaction. 

•  Automated  chemical  plants,  like  automated  weapons,  could  malfunction  and  cause  fire  or 
explosion  and  the  destruction  of  the  plant/weapon  itseK  There  is  a  greater  parallel  here  than 
with  banking.  However,  chemical  plants  are  not  designed  to  destroy  and  kill;  they  are  designed 
to  be  cost  effective  and  safely  and  accurately  process  products  from  raw  materials. 

These  examples  illustrate  two  points.  Weapon  ^sterns  have  some  features  in  common  with 
automated  commercial  systems  such  as  banking  and  chemical  plants,  but  their  purpose  is 
radically  different.  Design  and  purpose  are  not  easily  separated,  and  design  differences  must  be 
taken  into  account.  Additional  criteria  apply  to  the  procurement  of  weapons.  Failimeofa 
^stem  in  combat  cannot  be  recovered  by  suing  the  vendor  who  sold  the  feulty  product,  or 
arresting  and  jailing  the  hacker  who  sent  the  system  awry.  A  malfunctioning  banldr^  system  or 
chemical  plant  can  be  shut  down  or  taken  off-line,  creating  only  a  temporary  inconvenience  to 
the  customer  imtil  it  is  properly  working  again.  A  system  that  malfunctions  in  combat  requires 
an  entirely  different  response  from  its  operators  and  can  result  in  fer  more  ffian  an  inconvenience 
to  those  depending  upon  it. 

An  additipTial  difference  in  commercial  and  military  systems  is  the  concept  of  “battleshort.” 

That  k,  the  capability  to  continue  operation,  or  start  operation,  despite  equipment  status 
anomalies  that  would  otherwise  render  the  equipment  unavailable.  Battleshort  bypasses  safety 

*  Throughout  this  report  the  term  “computer  program”  means  executable  code  in  the  form  of 
firmware  and  software  stored  in  a  readable  media.  The  term  “software”  is  used  to  mean  all  other 
forms  of  software  including  source  code. 
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interlocks  and  equipment  protective  devices  except  those  whose  bypassing  would  cause 
immediate  and  serious  casualty  in  the  event  of  a  feult  condition.  Battleshort  is  an  abnormal 
mode  of  operation  used  in  emergencies  and  is  a  design  feature  of  selected  systems.  In  essence, 
battleshort  provides  the  option  of  allowing  a  system  to  potentially  incur  limited  damage  to 
prevent  a  greater  loss. 

Acquisition  Reform  advocates  the  use  of  commercial  products  in  military  systems  to  the 
maximum  extent  possible.  The  consumer  marketplace  is  such  that  producers  of  commercial 
items  strive  to  ensure  that  products  are  safe  to  use  and  “fail  safe”  when  they  malfunction. 
Vendors  accept  responsibfl^  for  their  products  and  “certify”  their  use  by  private  consumers  by 
means  of  a  guarantee.  However,  this  does  not  mean  that  systems  constructed  from  commercial 
parts  will,  by  the  vendor  guarantees,  be  certified  as  “mission  ready.”  There  is  nothing  inherent 
in  the  Acquisition  Reform  approach  that  ensures  that  the  risk  in  using  commercially  based 
military  systems  is  known,  understood,  and  minimized.  The  assurance  that  a  vendor  will  replace 
a  product  that  fails  under  normal  use  cannot  be  taken  as  confidence  that  the  product  can  be 
'  depended  upon  in  combat.  Commercial  vendors  strive  to  make  their  products  dependable  for 
their  intended  use,  but  no  commercial  vendor  can  be  held  liable  for  products  bought  by  the 
federal  government  and  used  in  warfore.  Any  approach  used  to  acquire  products  for  use  in 
warfere  must  acknowledge  the  federal  government’s  sole  responsibility  to  ensure  the  public  trust. 
The  successful  application  of  Acquisition  Reform  requires  that  a  core  technical  process  be 
established  that  leads  to  commercialfy  based  systems  that  are  certified  for  use  in  warfere. 

Certifying  the  behavior  of  a  system  means  that  the  ride  in  using  it  has  been  qualified.  Qualifying 
risk  requires  the  following: 

•  Defining  system  risk  criteria 

•  Verifying  the  system  meets  the  definition 

•  Testifying  to  the  fidelity  of  the  verification 

The  first,  defining  risk  criteria,  means  specifying  an  acceptable  requirement  or  standard  that  can 
be  measured.  The  second,  verification,  means  determining  that  design  data  and  test  data  are 
accurate  and  consistent  with  the  documented  standard.  The  third,  testifying,  means  the  human 
act  of  attesting  by  report,  letter,  or  other  means  that  the  system  has  met  the  standard  within 
specified  limits.  The  act  of  certifying  includes  the  acceptance  of  accountability  and  liability  for 
the  deployed  system  and  responsibility  for  the  certification  process,  including  the  standard 
criteria  used.  Certification  may  include  a  statement  of  known  capability  and  limitations. 
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3.0  RISK  DEFINITION  AND  QUALIFICATION 


The  acquisition  process  requires  “risk  management”  to  control  cost  and  schedule  and  ensure 
success  in  meeting  procurement  milestones.  Procurement  “risk”  is  controlled  by  an 
administrative  process.  Throughout  this  report,  the  term  “risk”  applies  to  physical  events  that  are 
controlled  in  the  core  technical  process. 

As  previously  defined,  a  computer  automated  system  is  a  machine  consisting  of  two  or  more 
physically  and  functionally  related  parts  that  are  interconnected  and  convert  iiq)ut  to  output.  If  a 
malfunction  occurs  somewhere  in  the  system,  then  the  system  may  M  by  providing  an  incorrect 
output  or  no  output.  Also,  if  the  system  contains  moving  parts  powered  by  electrical, 
mechanical,  or  chemical  energy,  then  a  malfunction  could  cause  the  system  to  damage  itself  or 
injure  or  kill  those  operating  it.  The  risk  in  using  any  system  is  that  it  may  fail  to  operate  as 
ejqpected.  Military  systems  are  expected  to  meet  an  operational  requirement  and  are  used  to 
support  the  accon:q>lishment  of  a  specific  mission.  The  risk  in  using  a  system  in  warfare  is  the 
risk  that  the  mission  may  not  be  accomplished  if  the  system  foils  to  operate  as  expected.  If  the 
system  “foils  safe”  then  it  may  be  possible  to  restore  it  to  proper  operation  and  still  accomplish 
the  mission.  If  the  system  cannot  be  quickly  restored,  or  the  foilure  results  m  a  hazard,  the 
mission  may  foiL  It  follows  that  the  risk  in  using  a  system  in  warfare  is  that  failure  of  the 
system  to  work  as  e^cted  may  lead  to  failure  to  accongrlish  the  mission.  The  consequence  of 
this  risk  is  the  possibility  of  a  hazard  caused  by  the  foiled  system  itself  and  the  prospect  of  loss  of 
life  and  property  by  enemy  (or  friendly)  action  due  to  the  mission  having  foiled. 

Qualifying  the  risk  of  using  computer  automated  systems  in  warfare  involves  complex  and 
interrelated  issues.  Risk  assessment  is  predicated  on  failure.  It  must  be  assumed  that  all 
systems  can  and  will  foil. 

•  Hardware  rnmpnnents  will  wear  out  and  foil  due  to  material  defects  and  age. 

•  Computer  programs  will  have  foults  due  to  the  huge  number  of  sequences  that  are 

possible. 

In  addition  to  these  foilure  modes,  complex  systems  may  also  have  design  defects  that  escape  the 
most  careful  development  process  and  appear  unejq)ectedly  after  the  system  is  deployed.  And, 
cause  and  effect  relationships  are  not  always  completely  imderstood,  so  fixes  may  not  have 
addressed  the  true  cause  of  the  defect.  Computer  programs  used  in  deployed  systems  contain  so 
many  interrelationships  that  defects  are  likely  to  be  found  during  use.  Most  computer  programs 
undergo  almost  continuous  improvement,  and  this  insidious  nature  of  software  requires  foult 
tolerant  designs  and  diligent  quality  assurance. 

The  consequence  of  a  failure  occurring  somewhere  within  a  system  is  that  it  may  lead  to  a  safety 
hazard  or  result  m  mission  negation.  The  issue  of  safety  will  be  discussed  first  since  it  is  also 
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inrpnrtflTit  in  the  nonoperational  activities  of  system  development,  testing,  and  training.  The 
more  complex  problem  of  mission  negation  will  be  addressed  second. 


3.1  SAFETY  RISK 

A  system  that  suffers  a  malfunction  while  operating  may  present  a  safety  hazard.  For  example, 
an  electrical  short  that  results  in  a  jBre  or  a  software  design  flaw  or  iii5)lementation  defect  that 
causes  a  control  computer  to  send  the  wrong  signal  resulting  in  either  unintended  action  or  lack 
of  intended  action.  TTie  severity  of  the  safety  hazard  and  the  probability  that  it  will  occur  defines 
the  risk  in  using  the  system  Hazard  severity  is  typically  described  as  shown  in  Table  1 .  The 
probability,  or  frequency  of  occurrence,  of  a  malftmction  leading  to  a  hazard  may  range  firom 
fi-equent  to  improbable.  Ideally,  probability  and  severity  would  appear  as  shown  notionaJfy  in 
Figure  2.  It  should  be  kept  in  mind  that,  for  weapons  and  particularly  nuclear  weapons,  there  are 
severity  levels  above  catastrophic  if  the  effects  of  detonated  warheads  are  included.  It  should 
-also  be  kept  in  mind  that  a  system  does  not  have  to  malfunction  to  present  a  safety  hazard.  For 
example,  most  radars  present  a  microwave  radiation  hazard  to  people.  And  weapon  systems  are 
dangerous  by  their  nature  and  their  design  requires  that  methodical  consideration  be  given  to  the 
safety  of  operators  and  bystanders. 

TABLE  1.  HAZARD  SEVERITY 


Hazard  Severity 

Effect  on  People 

Effect  on  System 

Negligible 

No  injury 

No  Damage 

Marginal 

Minor  injury 

Minor  Damage 

Critical 

Severe  injury 

Major  Damage 

Catastrophic 

Death 

Destruction 

Methods  for  evaluating  and  managing  system  safety  are  well  known  [4,  5].  The  process  begins 
with  the  first  steps  in  design,  and  a  disciplined  analysis  is  conducted  diroughout  the 
development.  Many  safety  issues  are  straightforward,  such  as  use  of  toxic  materials  and  proper 
electrical  grounding.  The  latter  is  different  for  shipboard  applications  in  that  ships  do  not  enjoy 
the  earth  groxmd  for  which  most  commercial  systems  are  designed.  Safety  analysis  may  lead  to 
design  changes  to  eliminate  or  reduce  the  possibility  of  hazards.  Any  part  of  a  system  that  has 
the  potential  of  becoming  a  hazard  is  considered  a  safety  critical  part.  These  safety  critical  parts 
should  be  identified,  and  attention  given  to  preventing  accidents.  Military  systems  are  inherently 
dangerous  and  rigorous  operator  training  is  necessary  to  prevent  accidents.  Weapon  systems  by 
their  very  nature  will  have  conponents  that  are  safety  critical  and,  where  possible,  these 
components  should  be  isolated  fi'om  the  rest  of  the  system  to  prevent  malfunctions  that  occur  in 
parts  that  are  not  safety  critical  firom  feeding  through  to  parts  that  are  safety  critical. 
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Probability  of 
Occurrence 


Hazard  Severity 

FIGURE  2.  HAZARD  SEVERITY  VERSUS  PROBABILITY  OF  OCCURRENCE  (NOTIONAL  CURVE) 

An  ftvaiTiple  of  weapon  system  safety  design  is  the  way  engagement  orders  are  handled  by  the 
Aegis  Weapon  System  in  the  AN/UYK-43  baselines.  Air  target  tracks  are  managed  in  the 
system  by  assigning  them  track  numbers.  The  Command  and  Decision  computer  manages  all 
tracks  and  is  used  to  initiate  an  engagement.  When  an  operator  orders  an  eng^ement  of  a 
designated  track,  a  message  to  engage  a  specific  track  is  sent  from  the  Command  and  Decision 
computer  to  the  Weapon  Control  conqjuter,  which  in  turn  prepares  a  missile  to  be  launched  to 
intercept  the  designated  track.  The  safety  question  that  arose  was  “What  if  a  software  fedt 
caused  the  wrong  track  number  to  be  specified  the  conputers?”  To  reduce  the  possibility  of 
firing  at  the  wrong  target,  the  engage  order  sent  to  the  Weapon  Control  con^iuter  is  repeated 
back  to  the  Command  and  Decision  conqiuter  for  verification.  If  both  computers  do  not  get  the 
same  answer  twice,  the  engagement  is  aborted.  As  a  final  measure,  the  operator  can  abort  the 
engagement  after  the  missile  is  laimched  by  sending  a  microwave  signal  from  the  ship  to  the 
miasilft  rfliising  the  warhead  to  explode  before  reaching  the  target.  This  design  approach 
acconplished  the  following  three  things. 

•  It  reduced  the  possibility  of  ei^aging  the  wrong  target. 

•  It  everyone  involved — civilian  engineers  and  sailors — ^aware  of  the  potential  danger 

of  engaging  the  wrong  target. 

•  It  gave  the  operators  confidence  in  using  the  system. 
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This  example  was  used  to  illustrate  a  safety  issue.  However,  there  are  mission  aspects  as  well. 

A  system  used  in  warfere  that  malfunctions  in  a  way  that  leads  to  engaging  the  wrong  target 
coidd  lead  to  mission  failure.  In  general,  any  safety  problem  suggests  a  mission  problem  as  well. 


3.2  MISSION  RISK 

This  discussion  of  mission  risk  assumes  that  the  hazard  severity  is  negligible  as  listed  Table  1 .  If 
success  of  a  mission  depends  on  a  given  system,  then  that  system  must  be  available  and  reliable; 
when  called  upon  it  must  work  and  work  as  e5q)ected.  Mission  risk  is  determined  by  availability, 
reliability,  and  maintainability  of  those  systems  that  are  depended  upon  to  accomplish  the 
mission.  Mission  risk  is  determined  by  how  resilient  the  ^stem  is  to  malfunctions  and  by  how 
long  it  takes  to  correct  a  malfunction  and  restore  the  system.  Mission  risk  must  also  include  the 
relationship  of  the  system  to  the  mission.  Systems  have  traditionally  been  placed  in  two 
categories:  mission  critical  and  not  mission  critical.  For  example,  for  surfece  combatants, 

-  mobility  is  mission  critical,  but  an  automated  bridge  for  steerage  and  engine  control  is  not 
considered  mission  critical  if  a  manual  backup  is  provided.  Weapon  systems  are  typically 
identified  as  mission  critical  since  manual  backup  is  impossible.  Mission  critical  systems  receive 
more  attention  to  detail  than  those  that  are  not  mission  critical. 

Traditionally,  mission  critical  systems  had  to  meet  more  stringent  requirements,  cost  more,  and 
as  a  result,  systems  designated  as  mission  critical  were  held  to  a  minimum.  As  systems  become 
more  integrated  and  complex,  the  line  between  critical  and  not  critical  becomes  blurred.  Time  is 
a  key  fiictor  in  mission  criticality.  For  exan^le,  a  Tomahawk  mission  may  take  an  hour  or  more 
to  execute  so  that  10  minutes  spent  in  correcting  a  malfunction  may  not  affect  the  outcome  of  the 
mission.  But,  in  contrast,  an  air  defense  mission  may  last  only  a  few  minutes,  and  there  may  be 
no  time  available  to  correct  malfimctions.  Thus,  systems  may  be  equally  mission  critical  but 
have  different  requirements  for  availability  and  for  time  to  repair.  To  evaluate  mission  risk, 
three  questions  about  the  system  and  its  parts  must  be  answered. 

•  What  effect  would  feilure  have  on  the  mission? 

•  How  often  can  feilure  be  ejqjected? 

•  How  long  does  it  take  to  restore  the  system  after  a  failure? 

Analysis  to  answer  these  questions  begins  with  the  concept  of  operation  of  the  system  and 
continues  as  the  system  design  progresses  through  production  and  delivery. 

Mission  risk  analysis  should  be  done  for  the  system  in  the  context  of  the  ship  and  an  acceptable 
value  for  the  probability  of  mission  feilure  determined.  The  greater  the  potential  loss,  the 
smaller  the  desired  probabihty  of  incurring  it,  so  that. 

Probability  of  mission  failure  (Pm)  ~  1/value  of  the  loss. 

The  potential  loss  could  vary  from  minor  damage  to  the  ship  to  major  damage  and  casualties  and 
ultimately  to  loss  of  the  ship  with  in:^)act  on  the  mission  of  the  deployed  force.  It  must  be  kept  in 
minH  that  the  cause  of  mission  failure  discussed  here  is  due  to  system  malfunction  and  not  to 
enemy  action.  A  system  malfunction  that  increases  vulnerability  to  enemy  action  due  to  mission 
feilure  can  lead  to  potential  loss.  In  some  cases  it  may  be  possible  to  restore  a  failed  system 
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&steners  prevented  the  17,000  ton  Iwo  Jima  from  performing  its  mission.  Attention  to  detail  is  a 
key  ingredient  in  qualifying  risk  and  ensuring  that  systems  work  as  e}q)ected. 

This  example  also  illustrates  how  illusive  certification  can  be.  The  incorrect  nuts  and  bolts  that 
were  used  clearly  satisfied  form,  fit,  and  function  and  thus  appeared  to  be  valid  replacement 

parts.  When  installed,  the  festeners  were  seen  to  be  consistent  with  the  system  design.  But  they 

were  not  consistent  with  the  elevated  teirqDerature  and  pressure  environment  in  which  the  system 
was  intended  to  operate.  Form,  fit,  and  function  must  be  satisfied  in  the  intended  operating 
environment. 

In  this  example  the  system  did  not  M  due  to  a  defect  in  the  original  design  but  due  to  a  defect 
that  was  introduced  during  maintenance,  which  illustrates  that  risk  quaMcation  is  not  a  one  time 
event  but  continues  throughout  the  life  of  the  system.  The  original  certification  has  to  be  ^ 
preserved,  and  system  maintenance  and  repair  procedures  included  as  part  of  the  overall  risk 
qualification  process.  Conqmter  programs  are  insidious  in  this  regard  as  they  can  and  do  wntain 
-latent  defects— ‘fougs”  that  may  be  represented  by  only  one  or  two  bits  in  programs  contain^ 
millions  of  bits.  Some  “bugs”  will  be  of  little  consequence  while  others  may  have  serious  side 
effects.  “Bugs”  will  be  triggered  by  a  unique  combination  of  events  and  occur  ui^redictably 
ttnring  tests  and  during  routine  use  of  the  system.  Once  found  they  can  be  eliminated  by 
changes  to  the  computer  program.  Unlike  the  Iwo  Jima  exan^le,  maintenance  of  the  conqiuter 
program  does  not  rely  on  spare  parts  but  requires  continuously  available  support  to  diagnose  and 
elirdnate  friults  to  preserve  certification. 
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4.0  TRADITIONAL  CERTIFICATION  PROCESS 


“Certification”  is  often  incorrectly  assumed  to  be  the  final  test  or  inspection  of  something 
followed  by  the  act  of  declaring  it  “certified.”  In  reality,  the  act  of  certifying  is  the  final  step  of  a 
complex  engineering  and  management  process.  The  traditional  process  of  approving  products 
and  systems  as  ready  for  Fleet  use  meant  that  there  was  confidence  that  the  product  or  system 
would  meet  prescribed  criteria.  The  required  confidence  was  gained  by  developing  standards 
and  continuously  measuring  products  against  them  This  involved,  for  example, 

•  Design  and  requirements  documentatioa 

•  Change  control 

•  Qualification  testing. 

•  Engineering  analysis  and  assessment. 

•  Qualified  risk  of  &ilure. 

Establishing  confidence  meant  that  the  product’s  behavior  was  consistent.  Knowledge  of  system 
behavior  was  derived  fi’om  understanding  its  internal  workings  and  fi’om  multilevel  test  and 
anafysis.  Availability,  reliability,  and  maintainability  were  quantified,  and  confidence  in  the 
system  ultimately  led  to  approval  for  Fleet  use. 

The  process  of  creating  certifiable  products,  that  is,  the  certification  process,  was  established  and 
institutionalized  over  the  decades  following  World  War  11.  A  tier  0  lustration  of  this  traditional 
acquisition  process  is  shown  in  Figure  3.  The  controls  that  were  established  to  maintain  the 
process  are  listed  in  Table  2.  This  list  is  by  no  means  exhaustive,  but  it  identifies  the  key 
controls  that  defined  the  prescribed  criteria  and  the  process  that  ensured  the  acquired  products 
met  those  criteria. 

The  process  shown  in  Figure  3  provided  two  things.  The  first  was  a  system  delivered  to  the 
Fleet.  The  second  was  repeat  production  of  system  parts  that  were  ussd  to  build  follow-on 
copies  of  the  system  and  for  spares  to  sustain  the  delivered  system  The  parts  included  hardware 
in  the  form  of  least  replaceable  units  (LRUs)  and  compvrter  programs  (CPs).  The  controls  over 
the  process  (Table  2)  ensured  that  the  LRUs  that  were  used  to  sustain  the  system  were  identical 
to  those  that  the  system  was  constructed  fi'om,  which  ensured  that  the  original  certification  of  the 
con^lete  system  was  not  invalidated  by  routine  maintenance.  In  most  cases,  LRUs  were  not 
elements  of  the  system.  Elements  of  the  ^stem  generally  consisted  of  several  LRUs  integrated 
together  to  form  a  unit  that  was  interfeced  with  other  elements  to  form  an  integrated  system 
LRUs  were  stockpiled  in  numbers  consistent  with  the  expected  usage  rate  so  that  failed  and 
damaged  ones  could  be  replaced  aboard  ship  as  routiue  maintenance.  The  nature  of  CPs  required 
them  to  be  sustained  by  a  logistics  process  difierent  than  LRUs. 
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FIGURE  3.  TRADITIONAL  SYSTEM  DEVELOPMENT  AND  PRODUCTION  PROCESS 


TABLE  2.  TRADITIONAL  ACQUISITION  PROCESS  CONTROL 


Qualified  Product  Lines 
Software  Qualified  with  GFI/GFE 
Full  Change  Control 
Full  Disclosure  and  Traceability 
Rigorous  Product  Testing 
Operational  Requirements 
Military  Standards 
Security  Classification 
Military  Specifications 
Military  Review  Boards 
Analysis 
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The  routine  replacement  of  CPs  aboard  ship  was  done  for  one  of  two  reasons: 

•  An  error  was  found  in  the  deployed  program. 

•  An  existing  function  was  changed. 

Replacement  was  made  by  installing  new  computer  programs,  or  by  installing  “patches”  to 
existing  shipboard  programs.  Corrections  to  firmware  required  replacing  the  defective 
component  aboard  ship  with  a  new  one.  A  different  approach  was  used  when  there  was  a  change 
to  one  or  more  system  elements.  This  was  a  conjunctive  change  of  hardware  and  software  and 
required  repeating  some  or  all  of  the  Multi-element  Integration  and  Test  (MBIT)  process  to 
ensure  compatibility  of  LRUs  and  CPs  and  to  recertify  the  system  For  this  traditional  approach, 
CPs  were  not  stockpiled  as  LRUs  were.  Rather,  CPs  were  “built”  as  needed  firom  source  code. 
Replacement  CPs  were  built  using  equipment  identical  to  that  used  in  their  original  development 
and,  before  delivery,  they  were  qualification  tested  on  con:5)uters  identical  to  the  ones  that  were 
used  in  the  shipboard  system  By  this  process,  changes  to  CPs  were  verified  and  the  original 
certification  of  the  complete  system  was  not  invalidated.  The  traditional  approach  allowed  LRUs 
and  CPs  to  be  maintained  independently  at  different  facilities,  although  in  the  actual  system 
'  aboard  ship  they  were  interdependent.  System  development  and  maintenance  were  coupled 
together  in  such  a  way  that  the  system  was  continually  certified. 

Acquisition  of  traditional  systems  required  a  major  effort  in  technology,  engineering,  and 
management.  The  core  technical  process  that  created  certifiable  systems  depended  on  control  of 
physical  risk  fectors  that  could  affect  performance,  safety,  maintenance,  and  training. 

Controlling  risk  meant  that  internal  Mure  modes  and  their  effects  on  system  behavior  had  to 
be  qualified.  For  exan:q)le,  understanding  the  risk  to  the  system  of  the  failure  of  a  given  LRU 
required  understanding  two  things: 

•  Possible  Mure  modes  of  the  LRU,  and 

•  The  role  of  the  LRU  in  the  system 

Both  required  detailed  knowledge  of  the  LRU  design  and  the  system  design.  If  the  LRU  is  a 
black  box,  that  is,  its  internal  design  is  invisible  and  imknown,  then  predicting  its  Mure  modes 
is  in^ssible.  If  the  LRU  is  a  black  box,  then  its  fimctions  will  not  be  visible,  its  role  in  the 
system  cannot  be  conq)letefy  understood,  and  the  risk  of  it  Ming  becomes  difficult  to  predict. 
The  function  of  the  system  is  the  collective  function  of  its  LRUs,  so  that  the  use  of  one  or  more 
black  box  LRUs  results  in  the  system  becoming  a  black  box.  That  is,  if  some  of  the  parts  of  the 
system  are  not  conqiletely  imderstood,  then  the  system  itself  cannot  be  completely  understood. 
The  end  result  in  using  black  boxes  is  that  the  severity  of  the  safety  hazard  and  the  probability  of 
mission  Mure  become  very  difficult  to  predict  or  control.  The  traditional  process  rejected  the 
use  of  black  boxes  and  required  design  disclosure  that  included  LRU  conqjonents  and  software. 
This  design  disclosure  was  used  for  two  purposes. 

•  Predict  and  control  risk,  and 

•  Configuration  control 

Predicting  risk  was  necessary  to  certifying  the  system,  and  controlling  its  configuration  was 
necessary  to  maintaining  that  certification. 

Consistency  was  a  mainstay  of  the  traditional  certification  process;  repeat  production  was  used  to 
ensure  interchangeability  of  parts  and  consistency  of  system  behavior.  Consistency  existed 
within  systems,  but  attempts  to  establish  consistency  across  systems  succeeded  onfy  on  a  small 
scale.  Independent  funding  lines  were  established  for  the  different  systems  so  that,  although  the 
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overaU  process  was  the  same,  the  LRUs  and  CPs  used  in  different  sy^ems  varied  in  de^  For 
traditional  systems,  interchangeability  in  the  horizontal  sense  was  limited  by  configuration 
control  in  the  vertical  sense.  The  result  was  that  system  certification  w^  tafiored  to  each  system 
and  consistency  across  systems  was  reflected  in  the  use  of  common  mihtary  standards  and 
militaiy  specifications.  The  inability  to  standardize  at  the  LRU  and  CP  level  across  systems 
constrained  interoperability. 
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5.0  ACQUISITION  REFORM  SYSTEM  DEVELOPMENT 


Adapting  the  traditional  core  technical  process  to  one  that  depends  on  commercial  business 
practice  poses  a  challenge.  The  next  generation  of  military  systems  is  ejspected  to  be  composed 
of  parts  purchased  from  the  private  sector  [10].  The  recommended  approach  is  that  commercial 
products  not  be  modified  to  satisfy  system  needs  but  that  system  requirements,  architecture,  and 
design  are  to  be  traded  off  so  that  commercial  products  can  be  used  off-the-shelf  [1 1].  For  most 
-  Navy  systems  there  are  some  requirements  that  cannot  be  satisfied  by  commercial  products.  For 
example,  neither  5in/54  gun  barrels  nor  Standard  Missile  midcourse  guidance  computer 
programs  are  available  commercially.  Systems  developed  under  the  Acquisition  Reform 
approach  will  consist  of  commercial  parts  (modified  or  unmodified)  and  Navy  parts  integrated 
together.  Commercial  products  may  be  changed  or  discontinued  in  response  to  sales  and 
competition  in  the  open  market  place.  Thus,  commercial  parts  for  maintenance  of  the  system 
and  production  of  follow-on  copies  of  the  system  may  not  be  the  same  as  those  used  in  design 
and  construction  of  the  lead  system.  It  will  be  necessary  to  continually  trade  off  system 
requirements,  architecture,  and  design  to  accommodate  changes  in  the  characteristics  of  the 
commercial  parts.  It  may  also  be  necessary  to  modify  the  Navy  unique  parts  as  well  to  make 
them  compatible  with  the  conamercial  parts.  The  end  result  is  that  systems  that  are  built  at  the 
rate  of  a  few  per  year  may  all  be  different  and,  once  put  into  service,  maintenance  may  continue 
to  change  each  one  in  a  different  way  as  spare  parts  are  acquired  over  time.  For  such  systems, 
the  risk  to  the  user  may  be  neither  known  nor  understood. 

The  overall  definition,  design,  and  development  of  the  system  is  traded  off  against  available 
commercial  parts.  Product  selection  requires  a  detailed  market  analysis  to  trade  off  requirements 
and  design  so  that  the  use  of  unique  parts  can  be  minimized.  The  ^stem  is  built  by  the  Multi- 
Element  Integration  and  Test  (MBIT)  process  that  combines  all  the  system  parts  and 
demonstrates  the  functional — ^input-output — ^behavior  of  the  complete  system.  Followii^  MBIT, 
the  system  can  vmdergo  additional  testing  before  it  is  approved  for  shipboard  use.  Success  in 
developing  the  system  will  depend  on  the  tradeoff  made  in  the  commercial  product  selection 
process.  The  tradeoff  process  requires  that  system  performance  (which  includes  availability  and 
reliability)  become  a  fimction  of  the  selected  products,  and  the  realizable  performance  may  be 
different  than  the  initial  desired  performance. 

A  system  integrated  with  commercial  parts  will  have  a  specific  configuration  determined  by  the 
tradeoflfe  made  in  the  parts  selection  and  design  process.  The  lifetime  of  commercial  products 
varies  and  can  be  as  short  as  1 8  months.  Given  this  volatility  of  available  parts,  the  specific 
configuration  of  a  system  will  be  time  dependent.  In  general,  repeat  production  will  not  be 
possible.  Indeed,  every  copy  of  the  system  may  differ  in  detail  from  every  other  copy.  Also, 
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system  support  requires  that  replacement  parts  be  available  for  the  lifetime  of  the  system.  This 
problem  of  rapidly  changing  characteristics  of  commercial  products  is  frequently  referred  to  as 
“technology  refresh.”  This  continual  introduction  of  “fresh”  products  to  replace  discontinued 
ones  leads  to  difficulty  in  development,  production,  and  support  of  military  systems.  Some  form 
of  replanning  and  reengineering  will  be  ongoing  throughout  the  life  of  the  system  [12]  leading  to 
incremental,  time  dependent  differences  among  all  copies  of  the  same  system.  Unless 
disciplined  engineering  and  management  steps  are  taken  to  minimize  configuration  (Merences, 
the  system  may  become  an  aberration  to  the  operators  who  are  ejq)ected  to  rely  on  it. 

Cost  has  been  a  major  factor  in  the  transition  to  the  reform  approach.  The  life  cycle  cost  of  the 
traditional  approach  has  been  viewed  as  prohibitive.  The  traditional  approach  is  characterized  as 
one  in  which  performance  is  the  independent  variable  and  cost  is  the  dependent  variable.  The 
reform  approach  advocates  the  converse;  cost  is  the  independent  variable.  The  traditional 
concept  is  to  ejqjend  fimds  to  develop  parts,  integrate  them  into  an  initial  system,  and  establish  a 
production  line  to  produce  copies  of  follow-on  systems  and  spares.  The  cost  curve  for  this 
.approach  is  shown  notionally  in  Figure  4.  The  reform  approach  is  to  reduce  initial  development 
and  production  costs  by  buying  commercial  parts  and  integrating  them  into  a  ^stem,  thus 
benefiting,  at  no  cost,  from  the  development  and  production  of  the  private  sector.  Although 
e}q)enditures  are  still  required  for  system  integration,  the  overall  cost  of  the  initial  system  is 
obviously  less  than  for  the  traditfonal  approach.  Unfortunately,  this  initial  system  c^ot  be  put 
into  production  at  no  cost,  and  integration  may  have  to  be  repeated  for  the  production  of  each 
foUow-on  system  since  there  will  be  changes  in  foUow-on  generations  of  off-the-shelf 
commercial  products.  The  cost  curve  for  this  approach  is  also  shown  notionally  in  Figure  4.  The 
reform  approach  clearfy  avoids  much  of  the  system  development  and  production  costs  at  the 
expense  of  almost  continuous  system  design  and  inte^ation.  The  reform  approach  clearly 
changes  the  quality  assurance  challenge  from  the  traditional  “build  it  right  the  first  time  and  then 
make  identical  copies”  to  the  reform  “buflt  it  right  every  time.” 

There  is  stark  contrast  between  the  traditional  and  the  reform  paradigms.  Ei^loring  these 
differences  will  guide  the  way  to  finding  a  new  core  technical  process.  The  differences  between 
the  two  that  are  key  to  developing  and  supporting  certifiable  ^ems  are  listed  in  Table  3  and 
explained  as  follows. 

•  Both  the  traditional  and  the  reform  paradigms  rely  on  MBIT  to  establish  the  system;  that 
is,  to  verify  the  overall  fimction  of  foe  system  and  to  qualify  the  parts — LRUs  and  CPs — 
as  true  parts  of  the  system 

•  Typically,  the  design  details  of  commercial  products  are  held  as  proprietary  to  prevent 
disclosure  to  competitors.  Using  proprietary  products  in  military  systems  means  that 
those  parts  of  the  system  must  be  treated  as  black  boxes.  Failure  analysis  regarding  these 
parts  will  require  perfecting  new  and  different  techniques,  or  design  disclosure  must  be 
negotiated  with  the  producer. 

*  The  Government  Accounting  Office  recommended  against  legislation  to  reduce  test  and 
evaluation,  because  it  “ . . .  would  undermine  a  key  management  control ...”  and  "...  test  and 
evaluation  should  increase,  not  decrease  ...”  (Testimony,  03/22/94,  GAO/T-NSIAD-94-124). 
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HGURE  4.  TRADITIONAL  AND  REFORM  LIFE  CYCLE  COST  (NOTIONAL  CURVES) 


TABLES.  ACQUISITION  PARADIGMS 


Traditional 

Reform 

MBIT  Established  Systran 

MBIT  Established  System 

Design  Disclosure  of  Parts 

Proprietary  Design  of  Parts 

Repeat  Production  of  Parts 

Aperiodic  Change  of  Parts 

System  Configuration 
Controlled 

System  Configuration 
Managed 

System  Requirements 

Specified 

System  Requirements 
Compromised 

Time 
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•  Once  system  development  is  completed,  repeat  production  of  parts  allows  for  a  suppfy  of 
identical  parts  for  repeat  production  of  the  system  and  for  spare  parts.  Commercial 
products  frequently  change  in  detail  design  as  well  as  overall  characteristics.  Since  there 
is  no  supply  of  identical  system  parts,  system  development  becomes  a  continuing  task 
and  MBIT  will  be  repeat^  to  reestablish  the  ^stem. 

•  Configuration  control  means  preserving  the  identity  of  the  system  by  requiring 
replacement  parts  to  be  the  same  as  the  parts  they  replaced  and  by  allowing  only 
preplanned  changes.  Proprietary  design  and  aperiodic  changes  make  configuration 
control  difiScult  or  inqwssible.  However,  the  coi^guration  can  be  managed  by  ke^i^  a 
detailed  record  of  current  parts  and  specific  characteristics  of  the  system  over  its  lifetime. 

•  Specifying  requirements  of  a  system  necessarily  specifies  its  parts.  If  parts  are  bor^t 
off-the-shel^  then  their  characteristics  may  not  exactly  match  those  required  to  satisfy 
predefined  system  requirements.  And  if  the  parts  cannot  be  modified,  then  the  predefined 
system  requirements  will  have  to  be  changed.  If  replacement  parts  differ  from  the 
ordinal  parts,  then  the  ^stem  requirements  must  again  be  changed  so  that  development 
requires  compromising  predefined  requirements,  and  sipport  may  require  continuing 
compromise. 

Given  these  general  characteristics  of  the  reform  process,  it  is  evident  that  traditional  system 
certification  is  no  longer  viable.  Efforts  to  cope  with  the  reform  process  continue  to  evolve  and 
the  need  for  certification  has  been  identified  [13].  However,  the  characteristics  of  tradition^ 
shipboard  systems — risk  to  the  user  is  known,  understood,  and  minimized — cannot  be  forfeited. 
The  extent  to  which  these  characteristics  can  be  achieved  will  now  be  discussed. 
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6.0  SYSTEM  CERTIFICATION 


Systems  developed  and  supported  by  traditional  methods  were  certified  as  a  consequence  of  the 
process  as  described  in  previous  sections  of  this  report.  Traditional  engineering  is  no  longer 
viable,  and  there  is  no  common  procedure  available  to  the  various  system  program  offices  in  the 
Navy.  A  new  process  should  be  defined  and  institutionalized  so  that  all  systems  will  benefit 
equally.  The  approach  taken  here  is  to  use  the  most  fimdamental  fectors  of  certification  as  a 
basis  for  reinventing  engineering  and  management  for  the  core  technical  process.  Figure  5 
provides  two  diflferent  interpretations  of  the  reform  process.  Figure  5a  represents  an  “open  loop” 
interpretation  wherein  products  are  delivered  directly  to  the  customer  bypassing  the  shore 
establishment.  This  interpretation  decouples  requirements  from  products  and  prevents 
certification  by  the  Navy  and  places  all  liability  on  private  industry.  This  approach  may  offer 
advantages  for  acquiring  products  requiring  warrantee  or  guarantee  but  not  for  systems  requiring 
certification.  The  “open  loop”  interpretation  will  not  be  discussed  further.  Figure  5b  reflects  a 
coupling  of  requirements  with  the  product  and  thus  “closes”  the  certification  loop  and  places  the 
burden  of  liability  on  the  Government  where  it  must  necessarily  be.  Not  shown  in  Figure  5b  is 
that  the  certification  recipient  is  the  commanding  officer  of  the  ship  receiving  the  system. 

The  certification  itself  should  explicit^  identify  any  known  limitations  of  the  system.  For 
example,  a  certification  could  state  that  the  system  has  no  known  faults  or  that  it  may  not 
function  properly  under  certain  specified  operating  conditions.  And  the  statement  tlmt  a  system 
is  not  well  imderstood  or  may  be  unsafe  and  should  not  be  operated  is  a  de  facto  certification. 

Certification  reflects  knowledge  of  the  system  and  how  it  is  maintained.  The  act  of  certification 
implicitly  states  that: 

•  The  intended  use  of  the  system  is  understood. 

•  The  environment  the  system  will  operate  in  is  known. 

•  The  system  design  is  consistent  with  the  operating  environment  and  intended  use. 

•  The  ^stem  was  implemented  consistent  with  its  design. 

•  The  capabilities  and  limitations  of  the  system  hxplementation  are  known. 

•  The  maintenance  process  will  preserve  the  design  implementation. 

•  Design  changes  (even  the  most  minor  ones)  will  require  recertification. 

•  The  system  was  properly  installed  aboard  ship. 

•  The  ship’s  crew  has  correct  and  sufficient  information  to  operate  and  maintain  the  system 
at  sea. 

Referring  again  to  Figure  5b,  the  engineering  and  management  process  must  provide  sufficient 
information  for  the  Navy  to  validate  the  nine  items  listed  above  and  discussed  below.  It  is  thus 
necessary  to  develop  an  acquisition  plan  that  ensures  validation. 
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HGURES.  TOP  LEVEL  PROCESS 

The  intended  use  of  the  system  is  understood.  This  requires  a  coirqjreheiisive  description  of 
the  system  performance,  its  mission,  and  how  it  will  be  operated  and  by  whom.  Here 
“performance”  is  taken  in  its  most  gaieral  meaning;  a  detailed  description  of  the  way  in  \^ch 
the  system  functions  when  in  use.  “Performance”  includes  such  things  as  operational  envelope, 
safety,  availability,  reliability,  and  maintainability.  Intended  use  of  the  system  is  nothing  less 
than  Government  furnished  information  specifying  the  desired  product. 

The  environment  the  system  will  operate  in  is  known.  Here  “environment”  includes  natural 
conditions  such  as  heat,  moisture,  and  vibration  as  well  as  the  effects  of  other  shipboard  systems 
and  people  that  the  desired  system  must  be  conpatible  with.  This  requires  that  the  desired 
system  analyzed,  not  in  isolation  but  embedded  in  the  natural  and  manmade  environment  it  is 
to  be  operated  in.  Products  from  this  analysis  include  some  performance  specifications  and  a 
Test  and  Evaluation  Master  Plan. 

The  system  design  is  consistent  with  the  operating  environment  and  intended  use.  This 
evaluation  consists  of  an  analysis  to  determine  if  the  design  has  taken  into  account  all  the 
performance  requirements  and  the  influence  of  the  operating  environment.  This  analysis 
requires  that  the  design  agent  furnished  information  that  can  be  used  by  the  Navy  to  review  and 
verify  that  the  design  satisfies  the  need.  In  cases  of  technical  difBciilty  or  high  cost,  it  may  be 
necessary  to  relax  performance  requirements  to  achieve  a  valid  design.  A  key  part  of  design 
analysis  is  evaluating  risk  as  discussed  in  Section  3. 
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The  system  was  implemented  consistent  with  its  design.  Inplementation  vaKdation  should 
proceed  in  parallel  with  the  implementation  so  that  validation  can  be  done  in  manageable  steps 
and  corrective  action  taken  when  necessary.  Implementation  is  not  a  replica  of  the  design  but 
rather  a  synthesis  of  physical  things  that  represents  the  design,  so  validation  cannot  be  achieved 
by  direct  comparison  of  the  real  with  the  abstract.  On  the  contrary,  it  is  necessary  to  determine  if 
the  implementation  satisfies  the  intent  of  the  design,  including  performance  and  risk  as  discussed 
in  Section  3.  Typically,  more  than  one  implementation  will  be  possible,  so  tradeoff  analysis  will 
be  required  to  select  the  best  approach.  Establishing  and  validating  an  optimum  implementation 
requires  an  intense  technical  effort  and  diligent  management. 

The  capabilities  and  limitations  of  the  system  implementation  are  known.  Predicting  the 
behavior  of  a  system  a  priori  is  generally  not  possible  since  it  depends  on  how  the 
inq)lementation  is  optimized.  It  is  inqwrtant  to  establish  the  “as  built”  characteristics  of  the 
system  and  demonstrate  its  performance  m  its  operating  environment.  A  Navy  Test  and 
Evaluation  Master  Plan  is  key  to  evaluating  the  operational  behavior  of  the  system  prior  to 
-  approval  for  Fleet  use. 

The  maintenance  process  will  preserve  the  design  implementation.  Maintenance  philosophy 
is  part  of  the  system  performance  requirement.  Details  of  the  maintenance  process  should  be 
established  during  design  implementation.  Shipboard  maintenance  will  involve  replacing  parts 
of  the  system  identified  as  LRUs  and  possibly  by  repairing  LRUs.  Conjputer  program 
maintenance  requires  a  shore  fecility  to  find  so^are  feults,  fix  them,  and  deliver  newly 
corrected  programs  to  ships.  The  procedures  used  for  maintenance  are  based  on  the  requirement 
that  the  operational  behavior  of  the  system  can  be  sustained  at  sea. 

Design  changes  (even  the  most  minor  ones)  will  require  recertification.  The  philosophy  for 
operation  and  maintenance  must  be  based  on  preserving  the  design  of  the  system.  The  “as  built” 
system  is  a  specific  configuration  of  specific  parts  approved  for  Fleet  use.  Any  change  to  the 
configuration  or  parts  introduces  an  unknown  that  could  cause  the  system  to  tehave  differently 
than  its  certified  behavior. 

The  system  was  properly  installed  aboard  ship.  Prior  to  deploying,  the  ship’s  commanding 
officer  needs  assurance  that  systems  have  been  properly  installed  and  are  functional.  The  shore 
establishment  that  is  re^nsible  for  delivering  a  specific  system  to  the  ship  must  supervise  the 
system  installation  and  checkout.  For  small  systems,  the  process  may  take  only  a  few  hours, 
while  for  complex  integrated  systems,  it  may  take  many  days.  In  the  case  of  external 
communication  links,  installation  may  involve  checking  ship-to-ship  connectivity. 

The  ship’s  crew  has  correct  and  suflicient  information  and  training  to  operate  and 
maintain  the  system  at  sea.  Operation  and  maintenance  of  the  system  requires  that  crewmen 
be  “certified”  by  approved  training.  This  requires  that  documents  and  procedures  used  by  the 
crew  accurately  reflect  the  ^stem  design,  system  implementation,  and  system  operation  and 
maintenance  philosophy.  This  requires  the  Navy  to  deliver  both  shipboard  and  classroom 
documentation  proven  to  meet  the  needs  of  the  ship’s  crew. 
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To  sustain  the  core  technical  process,  it  is  crucial  that  aU  nine  of  these  raplicit  fectors  be 
recognized  and  continuousfy  ensured.  Dedicated  and  dihgent  management  is  critical  to  success 
since  these  fectors  can  appear  as  in:q)ediments  to  schedule  and  cost  demands.  In  reality  they  are 
the  opposite.  Compromising  these  fectors  to  achieve  short  term  goals  can  lead  to  serious 
consequence.  One  exan^les  of  con^romise  is  the  readiness  of  USS  Hus  City  (CG  66)  and  USS 
Vicksburg  (CG  69)  to  deploy.*  Another  exanple  is  the  catastrophic  accident  aboard  the  space 
shuttle  Challenger.  Both  these  exanq)les  were  the  result  of  induced  instability  of  the  core 
t<yhnira1  process.  The  reasons  for  the  Challenger  accident  are  multiple  and  have  been 
documented,  but  they  boil  down  to  a  feilure  of  the  safety  certification  process.  Evaluation  of  the 
Challenger  core  technical  process  and  its  products  uncovered  safety  hazards  in  both  the  solid 
rockets  and  the  liquid  fuel  engine,  viiile  the  con:q)uter  system  was  determined  to  be  of  the 
highest  quality.  The  report  of  this  evaluation  [14]  offers  a  fundamental  perspective  on  foe 
meaning  of  certification  and  foe  roles  of  management  and  engineering  in  foe  core  technical 
process. 


6.1  THE  NEED  FOR  STANDARDS 

Certification  is  only  valid  if  referenced  to  a  standard.  Standards  are  foe  foundation  of  industry, 
trade  and  commerce  and  have  been  in  use  since  biblical  times.  Standards  provide  measures  of 
comparison  for  quantitative  or  qualitative  value;  criterion  that  defines  a  specific  condition  of  a 
commodity  or  human  behavior.  The  traditional  certification  process  used  Kfilitary  Standards  as 
discussed  in  section  4.0.  These  traditional  standards  vfoere  developed  specifically  for  foe 
purpose  of  procuring  and  using  products  in  the  military  environment.  Products  that  met 
identified  Military  Standards  were  de  fecto  certified.  Standards  used  for  commerci^  products 
generally  were  not  considered  adequate  for  products  intended  for  use  in  combat.  Military 
standards  were  needed  to  describe  the  operating  environment.  Military  Standards  were  also 
created  to  support  design  irrplementation  and  define  methods  for  production  and  testing. 

The  proliferation  of  standards  during  foe  past  century  lead  to  two  independent  sets— one  used  to 
procure  military  products  and  one  used  for  commercial  products.  The  two  sets  had  both 
similarities  and  differences.  For  example,  the  standards  required  for  components  used  in  foe  600 
psi  steam  plant  of  foe  USS  Iwo  Jima  (section  3.3)  were  similar  to  the  standards  for  conponents 
used  in  commercial  600  psi  steam  plants.  The  existence  of  an  equivalent  commercial  standard 
malfes  foe  use  of  a  separate  military  standard  difficult  to  justify.  However,  real  and  important 
differences  exist.  For  example,  foe  anticipated  shock  and  vibration  environment  for  warships  is 
more  sever  than  for  commercial  vessels.  Thus,  a  Military  Standard  is  required  for  shock  and 
vibration  regardless  of  the  availability  of  a  lesser  standard  for  commercial  products.  However, 
foe  most  striking  difference  in  commercial  and  militaiy  standards  is  purpose.  Commercial 
standards  are  necessary  to  fecilitate  commerce  and  are  driven  by  foe  market  place.  Military 
standards  are  driven  by  the  need  to  ensure  deployed  systems  work  as  expected.  Military 
standards,  in  general,  do  not  fecilitate  commerce  and  are  thus  viewed  as  inpeding  reform. 

*  Premature  installation  of  equipment  and  computer  programs  on  these  shps  degraded  their 
readiness  and  prevented  either  ship  fi'om  deploying.  Rework  to  make  these  ships  mission  ready 
took  over  a  year. 
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As  part  of  Acquisition  Reform,  the  defense  standardization  program  was  established  to  minimize 
the  use  of  government  unique  specifications.  Since  1994  thousands  of  standards  and 
specifications  have  been  canceled  and  thousands  of  others  inactivated  .  Some  Military 
Standards  have  been  replaced  by  commercial  standards  issued  by  the  Institute  of  Electronic  and 
Electrical  Engineers  (IEEE),  the  Standards  Engineering  Society  (SES),  and  the  American 
National  Standards  Institute  (ANSI).  Some  Military  Standards  have  been  converted  to 
handbooks  for  standard  practice  for  management.  For  example,  the  system  safety  standard  has 
evolved  into  a  standard  practice  to  be  chosen  by  contractors  as  need  [15]. 

Certifying  the  behavior  of  a  system  means  that  the  risk  in  using  it  has  been  qualified  against  a 
known,  valid,  documented  standard  that  accurately  defines  the  environment  the  system  is  to 
operate  in.  In  addition,  standards  for  testing  and  anafysis  must  be  applied  rigorously  . 

The  study  reported  here  did  not  include  a  detailed  investigated  of  specific  requirements  for 
certification  standards  for  shipboard  systems.  The  adequacy  of  the  present  residue  of  Military 
-  Standards  and  available  commercial  standards  to  support  shipboard  system  certification  was  not 
determined  as  part  of  the  study  reported  here. 

6.2  ROLE  OF  TEST  AND  ANALYSIS 

Testing  of  system  parts  and  the  complete  system  is  necessary  but  not  sufficient  to  fiilfy^  qualify 
the  system.  Full  qualification  requires  analysis  of  design  data  and  data  fi'om  tests  of  system  parts 
and  the  coirq)lete  system,  provided  the  data  is  of  adequate  fidelity.  Confidence  to  qualify  a 
system  is  developed  over  time  by  an  accumulation  of  analysis  of  design  data  and  test  data. 

The  process  of  ^stem  development  can  be  xmderstood  in  terms  of  three  stages  as  shown  in 
Figure  6.  The  process  starts  with  system  level  requirements  such  a  Mission  Needs  Statement 
followed  by  lower  level  and  more  specific  requirements  that  guide  the  design  down  to  the  point 
of  identifying  system  parts.  The  process  is  then  reversed  for  implementation  as  integration  and 
test  proceeds  upward  from  the  parts  level  to  the  element  level  until  the  complete  system  is 
created  and  operationally  tested.^  In  reality  this  description  is  idealized  in  tMt  the  actual  process 
is  not  a  continuous  smooth  flow,  but  interruptive,  iterative,  and  heuristic  as  technical, 
management,  and  budget  problems  appear  and  are  resolved.  The  end  result  of  this  requirements- 
design-integrate-test  process  is  a  system  consisting  of  a  complex  of  integrated  parts. 


*  See  the  Defense  Standardization  Program  web  page  at  http://dsp.dla.mil/ 


**  On  September  23, 1999  a  NASA  probe,  the  Mars  Climate  Orbiter,  foiled  to  enter  a  stable  orbit 
and  apparently  crashed  into  the  planet  by  mistake.  A  contributor  to  this  foilure  was  later  found 
to  be  confusion  over  the  standard  imit  of  measure — ^English  or  metric — ^that  was  used  for 
navigation. 

^  The  Government  Accounting  Office  concluded  that  this  process  is  successfiilly  being  followed 
by  the  leading  commercial  firms  they  visited  but  not  by  the  DoD  (Chapter  Report,  07/31/2000, 
GAO/NSIAD-00-199). 
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The  system  design  begins  with  high  level  requirements  and,  as  the  design  progresses  in  detail, 
the  requirements  e^q^and  in  detail  to  include  LRU  components.  The  test  and  integration  process 
starts  at  the  lowest  level  of  detail — components  of  the  LRU — ^and  progresses  upward  to  the  final 
integration  and  test  of  the  complete  system.  This  approach  of  top  down  requirements  and  design 
followed  by  bottom  up  design  inqjlementation  is  holistic  in  the  sense  that  the  system  as  a  whole 
is  defined  and  the  system  parts  can  be  viewed  in  the  context  of  the  whole.  It  also  provides  the 
best  cost  effective  results  because  the  implementation  proceeds  in  small  steps  so  that  design 
flaws  can  be  isolated  and  the  cost  of  fixing  them  will  be  contained.  The  solid  line  shown  in 
Figure  6  separates  two  engineering  and  management  domains  that  do  management  and 
engineering.  To  the  right  of  the  solid  line  is  the  domain  of  the  Navy  and  military  contractors. 

To  the  left  of  the  solid  line  is  the  domain  of  commercial  suppliers.  The  two  domains  have 
historically  been  independent.  The  requirements  and  design-integration-test  data  used  to  develop 
and  produce  commercial  products  typically  stay  within  the  commercial  domain.  Thus,  design 
data  and  test  data,  unless  disclosed  to  the  Navy  by  the  commercial  suppliers,  will  be  inconqjlete 
-and  analysis  of  the  system  will  be  deficient.  So  what  impact  does  this  analysis  deficiency  have 
on  quali^dng  the  system  so  it  can  be  certified?  The  answer  to  this  question  depends  upon 
qualifying  risk.  If  risk  can  be  qualified  to  an  acceptable  level,  then  there  is  no  impact  on 
certi^ing  the  system  If  not,  then  approving  the  system  for  Fleet  use  may  be  inqjrudent. 
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FIGURE  6.  STEPS  IN  SYSTEM  DEVELOPMENT 
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Qualifying  risk  in  using  a  ^stem,  as  e3q)lained  in  Section  3,  depends  on  several  interrelated 
fectors.  The  case  at  hand — a.  system  containing  nondisclosed  commercial  parts — ^is  the  classic 
“black  box”  problem.  To  illustrate  this  problem,  consider  Figure  7,  which  represents  a 
hypothetical  system  consisting  of  four  networked  elements.  Three  of  the  four  system  elements 
are  white  boxes  indicating  that  their  internal  workings  are  known.  The  fourth  element  is  a  black 
box  indicating  that  its  internal  workings  are  unknowiL  To  qualify  risk  it  is  necessary  to  perform 
an  analysis  for  each  element  to  determine  possible  ^ilme  modes,  possible  causes  of  feilure, 
possible  effects  of  feilures,  probability  of  occurrence,  and  criticality.  The  basis  for  this  analysis 
is  design  data  for  each  element,  such  as  functional  drawings,  schematics,  and  Allure  rates  of 
conqx)nents.  Since  there  is  no  design  disclosure  for  the  black  box,  it  must  be  treated  as  single 
device  that  converts  input  to  output  in  an  unknown  way.  The  criticality  of  this  element  to  the 
system  and  the  effect  of  its  Mure  on  the  system  can  be  analyzed,  but  the  Mure  modes,  possible 
causes  of  Mure,  and  probably  of  occurrence  are  difficult  to  determine.  The  element  could  be 
tested  to  determine  the  variance  in  ou^ut  versus  test  conditions  such  as  temperature,  vibration, 

.  input  variance  and  so  on.  This  approach  is  typically  how  the  probability  of  Mure  of 
components  such  as  transistors  is  determined.  However,  the  black  box  contains  multiple 
con^onents  that  must  be  treated  in  the  aggregate,  as  a  sirgle,  very  complicated  con:q)onent.  It 
will  be  necessary  to  test  several  copies  of  the  black  box,  which  will  introduce  yet  another 
variable;  that  is,  box-to-box  variance  due  to  con^onent  tolerance  and  the  manufecturing  process. 
The  primary  requirement  in  testing  is  to  get  a  large  enough  san^le  to  ensure  stationary  statistics. 
Confidence  in  test  results  requires  a  tradeoff  between  how  many  san:q)les  are  needed  and  how 
many  samples  can  be  collected  [16]. 


HGURE  ?.  HYPOTHETICAL  SYSTEM 


6.3  IMPACT  OF  ARCHITECTURE 

Systems  consist  of  two  or  more  interacting  parts  that  convert  input  to  output.  For  the  purpose  of 
this  discussion,  system  architecture  is  a  description  of  the  physical  relationship  of  the  parts  and 
how  the  parts  functionally  interact.  Unfortunately  this  view  of  architectxire  is  complex  since  it 
involves  both  hardware  and  software.  Hardware  contributes  to  system  architecture,  for  example, 
by  how  memory  is  accessed,  by  how  I/O  is  supported,  and  by  how  independent  processors  are 
interrelated.  Software  contributes  to  architecture  by  how  the  operating  system  works  and  by  how 
the  application  programs  relate  to  each  other  and  to  the  processors.  When  the  system  is 


28 


NSWCDD/TR-01/16 


operational,  the  hardware  and  software  conqwnents  interact  in  real  time  so  that,  in  to  view, 
system  architecture  is  not  static  but  dynamic  and  includes  time  dependency  of  functional 
relationships  of  the  system  parts.  System  architecture  can  be  conqilex  and  the  more  con^lex  it 

is  the  more  difScult  it  will  be  to  certify. 

Architectural  con?)lexity  will  be  determined  by  how  the  system  is  designed  and  how  the  design 
is  inqjlemented.  The  number  of  processors  and  how  they  are  interconnected  is  an  important 
fector.  The  simpler  the  interconnection,  the  sin5)ler  it  will  be  to  analyze  and  test.  Redundancy 
leads  to  conqjlexity.  For  exaitple,  a  system  consisting  of  N  parallel,  identical,  and  redundant 
elements  will  have  2^  modes  of  operation  including  the  degenerate  one  when  all  N  processors 
foil  Each  mode  will  in  fiict  represent  a  dififerent  implementation  of  the  system  and  will  require 
test  and  anafysis  to  validate  its  behavior.  If  the  same  system  used  nonidentical  elements,  then 
analysis  and  testing  would  become  even  more  difficult.  Thus,  design  m^lementations  c^ 
sinq)li£ied  by  limiting  redundancy  and  maximizing  commonality.  Limiting  redundancy  will  limit 
the  number  of  different  modes  of  the  system  requiring  certification.  Maximizing  commonality 
.  will  simplify  the  system  configuration,  minimize  documentation  and  analysis,  and  reduce  the 
required  test  environment.  On  the  other  hand,  redundancy  may  be  necessary  to  achieve 
availability  requirements  and  dissimilar  processors  may  be  needed  to  support  dififerent  types  of 
functions.  Thus,  architecture,  requirements,  and  certification  are  inseparably  related. 

A  large  part  of  the  system  architecture  is  determined  by  the  operating  system  and  how  it  manages 
resources  and  interfeces  with  the  application  programs.  Operating  systems  and  the  ^sterns  they 
are  hosted  on  fell  in  two  categories-n-eal  time  and  non  real  time.  Control  systems  are  real  time 
systems;  data  processing  systems  are  not  real  time  S3rstems.  Control  systems  are  required  to 
process  specific  tasks  at  specific  times  so  that  responses  are  neither  early  nor  late  but  periodic. 
Non  real  time  systems  are  used  to  process  aperiodic  tasks  such  as  graphical  user  interfece. 
Windows  is  a  commercial  operating  system  that  can  be  used  to  “run”  a  variety  of  non  real  time 
application  programs  such  as  WordPerfect.  VxWorks  is  an  example  of  a  commercial  red  time 
operating  system.  Most  traditional  weapon  systems  such  as  Aegis  and  Fleet  Ballistic  Missile  are 
real  time  systems.  The  architectural  differences  of  real  time  and  non  real  time  systems  are  as 
follows. 

•  In  real  time  systems,  tasks  “own”  the  hardware  resources  and  the  operating  system  is 
designed  to  manage  the  resources  (processors,  memory,  I/O)  and  support  task  execution. 
Task  priority  is  determined  by  the  tasks  and  stored  in  an  operating  system  queue.  These 
operating  systems  are  relative  small;  VxWorks  requires  about  0.80  MB  of  memory  to 
install 

•  In  non  real  time  systems  the  operating  system  “owns”  the  resources  and  is  designed  to 
service  tasks.  Task  priority  is  determined  by  the  operating  system,  which  also  manages 
the  system  resources  (processors,  memory,  I/O).  These  operating  systems  are  relatively 
large;  ^Mndows  requires  100  MB  or  more  of  memory  to  install. 

Large  operating  systems  pose  a  quality  assurance  challenge  due  to  their  shear  size.  Test  and 
analysis  to  eliminate  faults  in  operating  systems  such  as  Windows  or  UNIX  would  be  a 
TnoTiiimPinfol  task.  Small,  agile,  real  time  operating  systems  could  be  easify  quality  controlled 
and  system  certification  achieved.  The  current  AN/UYK-43  based  Aegis  Weapon  System  that 
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uses  a  0.250  MB  operating  system  has  been  fully  certified.  Aegis  and  other  sh^board  systems 
require  real  time  operation  for  only  part  of  the  tasks  they  perform.  Most  use  graphic  interfeces 
and  require  other  tasks  that  are  not  real  time.  Such  systems  are  combinations  of  real  time  and  not 
real  time  tasks  that  conplicate  the  architecture.  Collectively,  the  operating  system  and 
application  programs  represent  the  software  architecture  for  the  system.  Selection  of  the  correct 
operating  system  is  critical  to  successful  inq)lementation  of  a  system.  Much  of  the  overall 
behavior  of  a  system  is  determined  by  the  operating  system,  and  knowledge  of  its  internal 
workings  is  critical  to  qualifying  risk  in  using  the  system. 


6.4  ROADBLOCKS 

The  use  of  unmodified  commercial  products  in  shipboard  systems  introduces  roadblocks  to 
qualifying  risk  to  the  user  and  thus  prevents  establishing  and  maintaining  certification. 
Organization  and  acquisition  are  the  two  major  roadblocks  to  smooth  transitioning  to  a  new  core 
-  technical  process. 


6.4.1  Organizational  Roadblocks 

The  current  naval  shore  establishment  evolved  with  the  traditional  acquisition  process.  A 
significant  organization  feature  is  functional  segregation  that  resulted  from  the  feet  that  LRUs 
used  for  maintenance  were  identical  to  those  that  were  used  to  construct  the  system  and  that  CPs 
were  built  and  maintained  using  equipment  identical  to  that  used  in  their  original  development 
and  used  aboard  ship.  This  identity  of  parts  was  one  aspect  of  controlling  the  traditional 
acquisition  process  as  discussed  in  Section  4.  This  approach  allowed  the  functions  of 
development,  production,  and  logistics  to  be  segregated  and  shore  fecilities  tended  to  have 
limited  charters  and  rarely  provided  full  service  for  shipboard  systems.  Under  the  traditional 
approach,  the  engineering  fecilities  of  the  shore  establishment  evolved  into  organizations  that 
could  specialize  in  the  different  stages  of  the  process,  thus  improving  the  overall  efficiency.  The 
reform  approach  is  inconsistent  willi  this  infrastructure. 

Figures  8  shows  the  key  steps  in  the  traditional  approach  to  development,  production,  and 
siqjport.  The  system  proceeds  from  design  to  procurement  and  MBIT  of  the  lead  system,  which 
results  in  two  things:  (1)  a  production  model  of  the  system  that  can  be  evaluated  and,  (2) 
establishment  of  a  production  line  for  the  parts  of  the  system  and  auxiliary  equipment  needed  to 
produce  and  support  it.  Following  approval  for  Fleet  use,  the  production  lines  provide  products 
for  MBIT  of  foUow-on  copies  of  the  system  These  production  lines  also  provide  spare  parts  and 
auxiliary  equipment  for  support  (maintenance  and  repair)  of  the  systems  that  are  in  service  use. 
Configuration  management  of  a  qualified  product  line  allowed  software  support  to  be 
independent  of  hardware  support  except  when  changes  to  hardware  were  required.  This 
functional  independence  led  to  organizational  independence  of  the  hardware  and  software 
support  functions. 

A  new  approach  is  shown  in  Figure  9  that  is  derived  from  Figure  8  by  introducing  the 
Acquisition  Reform  requirement  to  use  commercial  products  to  the  extent  possible.  System 
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design  will  include  a  market  survey  of  commercial  vendors  to  determine  product  performance 
and  availability.  If  commercial  items  are  selected,  this  survey  will  continue  throughout  the  life 
of  the  system  to  assess  the  availability  of  the  commercial  items.  Decision  points  are  required  to 
deal  with  obsolescent  and  changing  commercial  items.  The  first,  as  shown  in  Figure  9,  is  the 
decision  for  procurement  and  MBIT  of  the  lead  system.  Changes  in  availability  of  one  or  more 
selected  commercial  items  at  this  point  means  that  the  system  cannot  be  implemented  consistent 
with  its  design  (see  Section  6.0)  and  the  design  must  be  revisited.  The  decision  to  produce 
follow-on  systems  depends  on  the  continued  availability  of  commercial  items  identical  to  those 
used  in  the  lead  system.  If  obsolescence  or  changes  have  occurred,  then  the  design  of  the  lead 
system  cannot  be  implemented  in  follow-on  systems  and  redesign  will  be  required.  Once  the 
system  is  in  use,  maintenance  and  repair  will  require  qualified  commercial  items.  Again 
obsolescence  or  changes  will  require  the  design  to  be  revisited  since  parts  substitutions  may  not 
preserve  the  original  design  implementation. 

In  comparing  Figures  8  and  9,  Figure  8  represents  a  linear  process  that  proceeds  in  steps  left:  to 
right  with  each  stage  (development,  production,  support)  leading  to  the  next.  Figure  9  represents 
a  nonlinear  process,  and  inputs  can  cause  the  process  to  reset.  The  development  stage  can  reset 
before  completion,  production  can  be  terminated  and  development  restarted,  and  maintenance 
can  trigger  redesign  and  restart  of  the  development  stage.  Figure  9  represents  a  process  in  which 
the  development  stage  sustains  both  the  production  and  support  stages.  In  Figure  8,  the  three 
stages  are  serial  and  can  be  managed  independently,  but  in  Figure  9,  production  and  support 
cannot  be  managed  independent  of  development.  Also,  Figure  8  represents  a  closed  process 
isolated  fi'om  outside  events  whereas  Figure  9  illustrates  an  open  process  that  can  be  influenced 
by  external  events — changes  in  the  commercial  market  place — ^that  can  trigger  instability. 
Overcoming  these  organizational  roadblocks  requires  the  following: 

•  Integrating  the  roles  of  development,  production,  and  support 

•  Buffering  external  inputs  to  the  process  by  controlling  the  rate  of  obsolescence  and 
change  of  commercial  items 


31 


NSWCDD/TR-01/16 


32 


FIGURE  8.  TRADITIONAL  DEVELOPMENT-PRODUCTION-SUPPORT  MODEL 
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FIGURE  9.  NEW  DEVELOPMENT-PRODUCTION-SUPPORT  MODEL 
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6.4.2  Acquisition  Roadblocks 

Short  lifetime,  nondisclosed  commercial  products  present  special  problems  in  engineering  and 
managing  the  new  core  technical  process,  and  both  are  roadblocks  to  qualifying  risk. 

Product  Lifetime  vs.  System  Lifetime.  Large  complex  systems  require  several  years  from 
concept  to  service  use  and  significant  upgrades  to  existing  systems  such  as  Aegis  may  take  as 
long  as  five  years.  Once  approved  for  service  use,  many  systems  will  be  deployed  for  10  years 
or  more.  Some  commercial  products  such  as  automobiles  and  aircraft  are  also  in  use  for  years 
and  are  supported  by  parts  manufactured  over  long  time  periods.  In  contrast,  commercial 
digital  electronics  and  software  tend  to  change  on  a  one-  to  two-year  basis  and  most  are  held  as 
proprietary  and  are  only  minimally  supported.  Systems  that  have  long  lifetimes  and  use 
commercial  products  that  are  supported  over  several  years  may  be  certifiable  based  on  historical 
performance  and  the  consistency  of  their  commercial  parts.  However,  systems  that  have  long 
lifetimes  and  use  commercial  products  that  have  short  lifetimes  and  limited  support  may  be 
diflScult  to  certify  due  to  uncertainties  in  their  design  and  configuration. 

If  the  components  used  to  implement  a  design  are  changed  after  the  implementation  is  verified,  it 
will  be  necessaiy  to  reverify  the  implementation.  Repeat  production  of  the  system  becomes 
difiScult,  and  each  copy  may  require  individual  design  implementation  verification.  Also,  spares 
used  to  support  the  system  may  be  different  than  the  original  parts;  thus  the  maintenance  process 
may  not  preserve  the  design  implementation.  The  result  of  using  short  lifetime  commercial 
products  in  systems  that  are  produced  over  time  is  that  all  copies  of  the  same  system  may  not  be 
the  same  and  routine  maintenance  may  introduce  more  changes  over  the  useful  fife  of  the  system. 
As  a  result,  each  copy  of  the  system  must  be  treated  as  potentially  unique.  For  example,  if  the 
system  is  to  be  deployed  on  N  ships,  then  it  may  be  necessary  to  provide  life  cycle  support  for  N 
variants  of  one  system  and  each  variant  would  have  its  own  certification  that  would  be  updated 
as  part  of  the  routine  maintenance  process.  Also,  variations  in  the  system  may  require  variations 
in  operator  training.  In  addition,  the  current  process  for  a  Battle  Group  workup  before 
deployment  requires  30  months  before  a  6  months  deployment.  If  this  D-30  process  exceeds  the 
lifetime  of  some  system  components,  then  there  will  be  additional  complications  in  certifying 
that  ships  are  ready  to  deploy.  The  use  of  short  lifetime  commercial  products  will  not  prevent 
certifying  systems  as  mission  ready,  but  the  process  may  lack  confidence  and  may  require 
operators  to  accept  more  risk  and  uncertainty. 

Nondisclosure  vs.  Disclosure.  Commercial  products  are  held  as  proprietary  to  prevent 
disclosure  to  competitors  so  that  design  data  and  source  code  is  generally  not  available  to  the 
Navy  for  system  risk  analysis.  Specific  knowledge  of  commercial  products  used  in  systems  is 
necessary  to  determine  if  the  system  was  implemented  consistent  with  its  design  and  to 
determine  the  capabilities  and  limitations  of  the  implementation.  Currently  there  is  no  practical 
method  that  will  allow  the  Navy  to  accept  and  protect  proprietary  data  for  commercial  products. 
Unless  data  rights  are  available  to  the  Navy,  it  will  be  necessaiy  to  deploy  systems  containing 
nondisclosed  products — ^black  boxes.  Testing  of  commercial  parts  both  in  and  out  of  the  system 
will  reduce  the  amount  of  disclosure  required.  Testing  can  be  used  to  determine  input-output 
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7.0  LEGISLATION  AND  THE  PUBLIC  TRUST 


The  legislative  responsibility  of  the  core  technical  process  is  manifest  in  the  feet  that  the 
recipient  of  the  act  of  certification  is  the  commanding  officer  of  a  ship.  The  government  is 
uhimatefy  responsible  to  its  people  for  the  actions  of  military  commanders  and  for  the  systems 
they  use;  certifying  shipboard  systems  is  an  integral  part  of  acquhing  them,  and  the  government 
is  accountable  for  ensuring  that  the  material  elements  of  an  afloat  command  are  mission  ready. 

The  Oath  of  Office  taken  all  commissioned  officers  is  to  “defend  the  Constitution  of  the 

United  States.”  And,  under  the  Constitution,  a  naval  officer  must  coirpfy  with  re^tions 
issued  by  the  Secretary  of  the  Navy  that  cover  all  aspects  of  naval  service.  Of  particular  interest 
are  the  following  sections  of  the  Navy  Regulations  [17]. 

•  Section  0702:  “Commanders  shall  be  reponsible ...  for  the  satisfectory  acconplishment 
of  the  mission  and  duties  assigned  to  their  command . . .” 

•  Section  0704;  “Commanders  shall  take  all  practicable  steps  to  maintain  their  commands 
in  a  state  of  readiness  to  perform  their  missions.” 

•  Section  0802:  “The  responsibility  of  the  commanding  officer  for  his  or  her  command  is 
absolute . . .  reponsibility  for  the  safety,  well-being,  and  efficiency  of  the  entire 
command.” 

These  Navy  Regulations  are  provided  for  under  Title  10,  U.  S.  Code,  Section  6011.  The  U.S. 
Code  consists  of  Acts  of  Congress  that  represent  the  will  of  tte  American  people.  Thus,  the 
public  provides  the  commander  of  a  warship  the  absolute  responsibility  for  readiness,  mission 
accomplishment,  and  the  safety  of  his  or  ho-  entire  command.  Responsibility  is  absolute  in  that 
it  cannot  be  delegated.  Failure  to  conply  with  Navy  Regulations  can  result  in  removal  fi'om 
command  and  possible  court  martial  as  set  for  under  the  Uniform  Code  of  Justice,  Title  10, 
Chapter  47. 

Making  a  commander  absolutefy  responsible  for  readiness,  mission  acconplishment,  and  safety 
means  that  the  shp  and  shipboard  systems  must  be  capable  of  working  as  expected.  No 
commander  could  legitimately  be  m^e  responsible  for  the  maintenance  and  operation  of  a 


‘  As  required  by  law  (U.S.  Code,  Title  5,  Section  3331,  Oath  of  Office),  the  same  oath  must  be 
signed  (Standard  Form  61,  Appointment  Affidavits)  by  all  individu;^  accepting  appointments  to 
the  civil  service. 
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poorly  constructed  ship  outfitted  with  defective  and  unsafe  systems.  The  credibility  of  the  Navy 
Regulations  and  Congressional  Legislation  depends  on  the  ship  and  shipboard  systems  being 
fully  capable  of  meeting  the  expectations  of  readiness,  mission  accomplishment,  and  safety.  The 
public  trust  bestowed  on  a  commanding  officer  represents  the  collective  will  of  all  the  people 
and,  by  virtue  of  his  or  her  assigned  responsibility,  the  afloat  commander  is  a  custodian  of  the 
public  trust.  Public  trust  in  the  Congressional  Legislation  that  assigns  the  commanding  officer 
absolute  responsible  for  maintaining  and  operating  shipboard  systems  implies  that  there  is  also 
public  trust  in  the  quality  of  those  systems.  Completeness  requires  an  absolute  responsibility  for 
qualifying  shipboard  systems  by  certifying  to  the  commanding  officer  that  they  are  mission 
ready  and  will  work  as  ejq)ected.  The  concepts  of  public  trust,  responsibility,  and  qualified 
systems  are  interdependent  and  inseparable. 

The  need  to  acquire  ships  “ . . .  capable  of  supporting  the  Navy’s  mission  fi’om  the  first  day  of 
commissioned  service ...”  is  expressed  by  OPNAV  Instruction  4700.8H  [18].  The  purpose  of 
this  instruction  is  to  “augment”  the  U.S.  Navy  Regulations.  It  assigns  responsibilities  to  various 
-  commands  and  defines  procedures  they  must  follow  for  the  legal  act  of  accepting  custody  of 
ships  delivered  by  private  contractors.  The  act  of  “accepting  custody”  of  the  ship  by  the  Navy 
requires  transition  firom  contractual  authority  to  command  authority;  that  is,  the  Navy’s  absolute 
responsibility  is  established  as  the  contractor’s  responsibility  is  brought  to  an  end.  The 
procedure  typically  takes  18  months  and  includes  a  6  months  warranty  period  following  delivery 
of  the  ship  in  which  the  contractor  is  held  financially  liable  for  failure  in  performance, 
workmanship  and/or  material  quality.*  The  Navy’s  responsibility  is  established  by  a  series  of 
“trials”  and  “inspections”  of  the  ship  and  all  of  its  systems  and  subsystems  to  ensure  the  ship  is 
complete  and  to  find  and  correct  “construction  deficiencies.”  Following  delivery,  the  ship  is 
commissioned  and  its  warfore  systems  undergo  trials  and  tests  at  sea.  The  ship  is  then  placed  in 
a  shipyard  to  correct  deficiencies  and  make  authorized  improvements. 

The  purpose  of  Instruction  4700.8H  is  to  make  sure  the  complete  ship  is  in  good  working  order 
by  macro  “trials”  and  “inspections”  of  complete  ^sterns;  visibility  into  design  is  not  required. 
This  instruction  assigns  command  responsibility  to  ensure  the  Fleet  has  “ . . .  complete  ships,  firee 
fi'om  both  contractor  and  government  responsible  deficiencies.”  The  procedme  is  intended  to 
uncover  operational  defects,  it  is  not  a  substitute  for  qualifying  risk.  It  provides  a  ship  level 
certification  process  and  takes  advantage  of  the  U.S.  Navy  Regulations  by  making  commanding 
officers  legally  responsible.  It  defines  the  apex  of  the  tot^  certification  process  but  does  not,  nor 
can,  define  the  lower  tier  reponsibilities  and  engineering  methods  that  should  be  used  in  the  core 
technical  process  that  creates  the  ship.  Command  responsibility  cannot  be  extended  to  the  core 
technical  process  so  long  as  it  requires  civilians  employed  by  the  Navy  and  by  private  companies 
under  contract  to  the  Navy.  And  responsibility  for  the  core  technical  process  has  been  further 
complicated  by  legislation  requiring  the  participation  of  commercial  vendors.  The  core  technical 
process  is  not  defined  in  Instruction  4700.8H.  But,  the  following  excerpts  taken  fi'om  paragraph 
7g  imply  that  a  process  to  qualify  risk  is  required. 


*  Title  10,  Section  2403,  requiring  major  weapon  system  warranties  was  repealed  in  November 
1997  following  a  General  Accounting  Office  report  (Chapter  Report,  06/28/96,  GAO/NSIAD- 
96-88). 
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•  “All  required  control  equipment,  auxiliaries,  fittings,  electronic  equipment,  combat 
system  equ^ment . . .  must  be  operable . . .  and  must  be  capable  of  meeting  performance 

specifications.” 

•  “Certification  of  sonar,  other  acoustic  processors,  combat  control  systems ...  is  required. 
When . . .  requirements  exist  which  cannot  be  achieved  until  after  delivery,  full 
certification  is  not  required.  However,  in  these  cases  all  other  elements  of  certification 
will  be  accomplished  and  certified  prior  to  Acceptance  Testing. 

•  “Conqilete  test  memoranda,  reports,  and  certificates . . .  must  be  available  for  inspection 
by  the  Trial  Board.” 

Current  legislation  does  not  appear  to  provide  regulations  for  assigning  responsibility  for  quality 
assurance  of  miHtary  systems.  The  U.S.  Code,  Title  41  (PubUc  Contocts),  Section  425,  appears 
to  prohibit  requirement  of  a  certification  by  a  contractor.  Indeed,  private  conqiames  that  contract 
■  for  Navy  ships  and  shipboard  systems  represent  not  all  but  only  a  small  fiaction  of  the  American 
people  and  thus  caimot  be  expected  to  be  custodians  of  the  public  trust.  The  warranties  and 
guarantees  typical  of  products  in  the  commercial  sector  establish  responsibilities  and  liabffities 
between  and  among  individuals  and  groups  but  they  are  meaningless  in  terms  of  the  public  trust. 
The  public  trust  requires  absolute  responsibility,  not  avenues  for  claims  under  commercial 
warranties  and  guarantees  that  would  seek  to  delegate  responsibility.  The  procurement 
philosophy  for  shipboard  systems,  particularly  weapons,  should  be  b^ed  on  the  concept  that  “all 
sales  are  final.”  Some  commercial  products  are  expected  to  meet  stringent  standards,  but  some, 
such  as  commercial  software,  are  not.  For  example,  in  early  1999  lobbyists  for  private  industry 
supported  Congress  in  establishing  and  passing  a  Y2K  liability  bill  (HR  775)  that  would  limit 
claims  in  the  private  sector  resulting  firom  computer  Mures  on  or  after  1  January  2000.  In 
contrast,  the  Chief  of  Naval  Operations,  acting  on  absolute  responsibility  for  his  command, 
established  a  Y2K  Project  under  Rear  Admiral  Jay  M.  Cohen,  USN,  who  in  turn  issued  a  Master 
Test  Plan  “ ...  to  assure  mission  functionality  of  all  operating  umts ...”  [19].  The  concern  of 
the  Navy  was  readiness  not  liability.  This  example  illustrates  how  the  two  domains — 
commercial  and  military— responded  differently  to  the  same  issue  even  though  both  relied  on 
Congressional  Legislation  to  define  responsibility.  The  key  difference  is  that  shipboard  systeim, 
whether  they  use  commercial  products  or  not,  should  carry  with  them  mission  ready  certification, 
not  limited  liability. 

There  are  mandatory  procedures  placed  on  major  defense  acquisition  programs  by  authority  of 
the  Secretary  of  Defense  in  accordance  with  Title  10,  U.  S.  Code.  The  revision  dated  January  4, 
2001,  Interim  Regulation,  DoD  5000.2-R,  [20]  reflects  an  acquisition  vision  of  the  core  technical 
process.  These  mandatory  requirements  are  not  presented  hierarchically;  they  must  all  be 
satisfied  simultaneously.  A  limited  review  of  this  document  from  the  perspective  of  mission 
ready  certification  finds  the  followii^  to  be  germane. 

*  Many  computer  programs  were  thought  to  contain  a  design  iinplementation  that  used  only  the 
last  two  digits  of  the  year  and  would  record  2000  as  00  and  thus  read  the  date  as  changing  fi'om 
1999  to  1900,  wreaking  havoc  with  military  systems,  government,  banks,  utilities,  and  the  public 
inJfrastructure  in  general. 
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•  Section  2.9. 1 .4.2:  “The  PM  [Program  Manager]  shall  work  with  the  user  to  defrae  and 
modify,  as  necessary,  requirements  to  fecilitate  the  use  of  commercial  and  non- 
developmental  items.” the  PM  shall  require  contractors  and  subcontractors  to 
use  commercial  and  non-development  items  to  the  maximum  extent  possible.” . . . 
“Preference  shall  be  first  to  commercial  items,  then  to  non-development  items.” 

•  Section  5.2.6.4:  “The  PM  shall  establish  formal  software  change  control  processes.” 

•  Section  5.2.8:  “The  PM  shall  establish  Reliability,  Availability,  and  Maintainability 
(RAM)  activities  early  in  the  acquisition  cycle.  The  PM  shall  develop  RAM  system 
requirements . . .  and  state  them  in  quantifiable,  operational  terms,  measurable  during 
development  and  operational  T&E.” 

•  Section  5.3.2:  “If  no  acceptable,  nongovernment  standards  exist,  the  Department  may 
define  an  exact  design  solution  with  military  specifications  and  standards,  as  a  last  resort, 
with  MDA-approved  waiver.” 

A  propensity  for  commercial  items  is  evident  although  the  traditional  need  for  change  control 
and  reliability,  availability,  and  maintainability  is  clearfy  stated.  But,  as  discussed  previously  in 
this  report,  neither  effective  change  control  nor  availability,  reliability,  and  maintainability  may 
be  satisfactorily  accon:q)lished  for  commercially  based  shipboard  systems. 

Fortunately,  Part  3  of  the  mandatory  procedures  for  PMs — DoD  5000.2-R — ^is  the  requirement 
for  a  Test  and  Evaluation  Master  Plan  (TEMP).  The  TEMP  and  its  contents  are  clearly  outlined, 
suggesting  a  traditional  approach  to  test  and  evaluation  that  specifically  includes  commercial 
items.  Also,  in  Section  3.5,  “The  developing  agencies  ...”  are  required  to  “Formally  certify  the 
system  ready  for  Operational  Test  and  Evaluation  (OT&E).”  Section  3.6  specifically  makes  the 
DoD  responsible  for  OT&E,  and  Section  3.6.2  limits  the  role  of  contractors  in  the  OT&E 
process.  Reliance  on  a  TEMP  and  the  DoD  responsibility  to  execute  it  should  fecilitate  mission 
ready  certification  of  shipboard  systems.  Although  T&E  is  a  large  part  of  qualifying  risk,  OT&E 
occurring  at  one  point  in  tin^  is  inadequate  to  ensure  availability,  reliability,  and  maintainability 
over  the  system  lifetime.  Control  over  and  visibility  into  the  parts  used  in  the  ^stem  are 
necessary  to  ensure  effective  change  control  and  reliability,  availability,  and  maintainability  over 
the  lifetime  of  the  system  Visibility  into  commercial  items  is  restricted  by  the  regulations, 
exceptions,  and  conditions  for  visibility  into  commercial  items  set  forth  in  the  U.S.  Code, 

Title  10,  Chapter  137,  Section  2320;  Rights  in  Technical  Data.  The  mandatory  use  of 
commercial  items,  and  their  use  as  black  boxes  as  is  implied  by  the  mandatory  procedures,  will 
reduce  the  effectiveness  of  OT&E  as  the  lone  method  for  certification  and,  over  the  life  cycle  of 
the  system,  introduce  risk  to  the  user. 
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8.0  CONCLUSION 


Certification  as  discussed  here  is  a  complex  and  protracted  process  that  ensures  the  integrity  of 
military  systems  that  must  be  operated  under  conditions  of  qioalified  risk.  There  will  be 
milestones  in  the  process  where  individuals  will  be  required  to  attest  or  approve  by  their 
signature  rHat  certain  work  has  been  completed.  Examples  are  requirements  documents,  test 
plans  and  reports,  and  reports  attesting  to  the  validity  of  implementation  of  hardware  ^ 
software.  Attesting  to  the  correction  of  known  conqjuter  program  design  flaws  is  an  inq)ortant 
duty  fi)r  fiompiiter  programmers  and  the  integrity  of  small  parts  of  the  system  may  be  attested  to 
by  workers  and  lower  level  managers  whereas  larger  system  parts  will  r^uire  the  suture  of 
higher  level  managers.  Once  the  system  reaches  approval  for  Fleet  use  its  certification  will 
consist  of  a  pyramid  of  signatures  each  reflecting  an  individual’s  sphere  of  responsibility.  The 
apex  of  the  pyranrid  will  be  the  final  certifying  official  for  the  complete  sy^em.  The  purpose  of 
identiJfying  individuals  in  the  certification  pyramid  is  to  establish  responsibility,  that  is,  by 
gigniTigj  the  employee  acknowledges  responsibility  to  the  employer  and  the  en:q)loyer 
acknowledges  responsibility  to  the  Fleet,  and  the  public  trust  is  assured. 

The  risk  in  ii«;ing  commercially  based  militaiy  systems  can  be  reduced  by  a  disciplined 
engineering  and  management  infirastructure.  Overcoming  acquisition  roadblocks  to  certification 
is  critical  to  ensuring  the  public  trust.  Organizational  changes  are  required  ivithin  the  naval  shore 
establishment  to  create  an  infrastructure  that  can  take  absolute  responsibility  for  the  qual^  of 
shipboard  systems.  New  business  practices  are  needed  to  form  teaming  arrangements  with  the 
shore  establishment,  military  suppliers,  and  commercial  suppliers  that  can  effectively  support  the 
absolute  responsibility  of  the  shore  establishment.  Altemative  methods  for  government  use  of 
and  protection  of  proprietary  products  and  intellectual  property  are  needed. 

Institutionalizing  a  new  core  technical  process  involves  management,  engineering,  and 
legislative  issues.  The  U.S.  code  and  the  DoD  acquisition  procedure  based  on  it  should  be 
reviewed  in  detail  and  modified  as  necessary  to  ensure  certification  requirements  are  supported. 
The  use  of  commercial  items  should  be  replaced  with  the  requiremrat  to  adapt 

commercially  developed  technology  for  military  use  consistent  with  Congressional  policy  set 
forth  in  Title  10,  Section  2501,  National  Security  Objectives  Concerning  National  Technology 
and  Industrial  Base. 

The  core  technical  process  should  not  be  used  to  fecilitate  legislation;  it  should  be  used  to  apply 
the  principles  of  physics  and  engineering  in  a  managed  acquisition  environment  protected  by 
legislation.  Mission  ready  certification  of  shipboard  systems  requires  dedicated  managers  ^d 
engineers  who  have  faith  in  the  process  and  products  for  which  they  have  absolute  responsibility. 
Faith  in  the  process  and  products  of  others  is  not  a  substitute.  The  current  process  will  evolve  to 
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flli£n  the  acquisition  vision  with  technical  reality.  Ultimate^  a  new  core  technical  process  will 
be  institutionalized  that  is  effective  in  making  sure  shipboard  systems  work  as  expected  and  are 
sustainable  at  sea. 

The  studies  reported  here  focused  on  ri^  qualification  and  certification  for  safety  critical  and 
mission  critical  systems;  particularly  software  dependent  systems.  Systems  used  ashore  for  other 
purposes  such  as  simulation,  analysis,  or  training  were  not  specifically  considered.  Nor  was  the 
use  of  commercial  products  m  hull  and  machinery  systems  such  as  electrical  and  piping.  The 
issue  of  electromagnetic  interference  was  not  addressed  although  it  is  a  growing  problem  due  to 
the  use  of  commercial  communication  equipment  aboard  ship.  Work  following  this  study 
should  address  certification  at  the  total  ship  level  and  encompass  hull,  machinery,  weapons,  and 
communications.  Specific  standards  for  certification  need  to  be  identified  and  applied 
rigorously. 
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